Oct 30 20:38:33.025012 ip-10-0-141-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:38:33.388206 ip-10-0-141-65 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:33.388206 ip-10-0-141-65 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:38:33.388206 ip-10-0-141-65 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:33.388206 ip-10-0-141-65 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:33.391288 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391198 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.391288 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391291 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391295 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391299 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391302 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391305 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391307 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391310 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391312 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391315 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391318 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391320 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391323 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391326 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391328 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391331 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391334 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391337 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391339 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391342 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391344 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.391368 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391347 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391349 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391352 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391354 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391357 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391360 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391363 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391366 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391377 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391388 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391392 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391394 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391397 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391400 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391402 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391405 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391408 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391411 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391414 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391417 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.391865 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391419 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391422 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391424 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391427 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391429 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391433 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391438 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391441 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391444 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391446 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391451 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391455 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391458 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391461 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391464 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391467 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391469 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391472 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391475 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.392322 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391477 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391480 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391483 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391485 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391488 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391491 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391493 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391496 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391499 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391502 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391504 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391507 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391509 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391512 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391515 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391518 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391520 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391523 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391525 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.392774 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391528 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391530 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391533 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391536 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391539 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391541 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391544 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391547 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391550 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391553 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391555 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391558 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391561 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391563 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391566 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391568 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391571 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391573 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391576 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391579 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.393218 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391581 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391584 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391586 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391589 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391592 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391594 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.391598 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392444 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392452 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392455 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392458 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392461 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392463 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392466 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392469 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392472 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392475 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392477 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392480 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392483 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.393702 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392486 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392509 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392513 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392516 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392519 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392523 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392525 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392528 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392533 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392536 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392538 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392541 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392544 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392549 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392551 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392554 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392556 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.394192 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392559 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392561 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392564 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392566 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392569 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392572 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392574 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392577 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392580 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392582 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392585 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392588 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392590 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392593 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392595 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392597 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392600 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392603 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392606 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392608 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.394646 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392611 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392613 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392616 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392621 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392623 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392626 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392628 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392631 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392633 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392635 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392638 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392640 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392642 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392645 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392647 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392650 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392668 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392671 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392673 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.395130 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392676 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392679 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392683 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392687 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392690 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392692 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392695 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392697 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392700 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392703 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392705 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392708 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392710 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392713 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392715 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392718 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392720 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392723 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392727 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.395614 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392731 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392735 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392739 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392743 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392746 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392749 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392753 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392755 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392758 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392761 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392764 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392766 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392769 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392772 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.392774 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393273 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393278 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393281 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393284 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.396100 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393286 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393289 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393291 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393294 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393297 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393299 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393302 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393305 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393307 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393310 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393312 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393316 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393319 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393322 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393324 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393327 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393329 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393332 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393334 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.396538 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393337 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393340 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393343 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393346 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393348 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393351 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393353 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393356 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393359 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393362 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393365 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393368 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393370 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393373 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393375 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393378 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393380 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393383 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393385 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393387 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.396993 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393390 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393392 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393395 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393397 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393400 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393402 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393405 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393407 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393411 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393415 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393418 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393421 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393426 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393429 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393432 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393434 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393437 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393439 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393442 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.397493 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393444 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393447 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393450 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393452 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393455 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393458 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393461 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393463 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393466 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393468 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393470 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393473 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393475 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393478 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393481 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393483 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393486 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393489 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393491 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393494 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.397987 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393496 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393499 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393501 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393504 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393506 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393510 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393514 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393518 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393521 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393524 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393526 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393529 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393531 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393534 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393536 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393539 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393541 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393545 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393547 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.398453 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393550 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393552 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393554 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393559 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.393561 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393677 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393688 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393698 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393703 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393708 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393712 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393716 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393721 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393724 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393727 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393731 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393734 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393737 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393742 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393746 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393749 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393752 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393755 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:38:33.398944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393760 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393763 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393766 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393769 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393772 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393776 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393779 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393782 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393787 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393790 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393793 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393796 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393801 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393804 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393809 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393812 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393815 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393817 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393821 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393824 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393828 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393831 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393834 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393837 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:38:33.399478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393840 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393844 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393847 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393853 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393856 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393859 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393862 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393865 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393868 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393870 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393873 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393876 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393880 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393883 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393886 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393889 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393892 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393896 2580 flags.go:64] FLAG: --help="false" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393899 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393902 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393906 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393909 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393912 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:38:33.400092 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393916 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393919 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393922 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393924 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393927 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393930 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393933 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393936 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393939 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393942 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393945 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393947 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393952 2580 flags.go:64] FLAG: --lock-file="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393955 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393958 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393960 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393966 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393969 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393972 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393975 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393978 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393981 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393984 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393987 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393992 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:38:33.401085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393995 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.393999 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394002 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394005 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394009 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394012 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394015 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394018 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394022 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394029 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394032 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394036 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394039 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394042 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394045 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394048 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394050 2580 flags.go:64] FLAG: --port="10250" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394053 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394056 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0f225ae7ec694b03a" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394061 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394064 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394069 2580 flags.go:64] FLAG: --register-node="true" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394074 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394079 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394084 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394088 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:38:33.402285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394091 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394095 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394098 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394101 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394104 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394107 2580 flags.go:64] FLAG: --runonce="false" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394109 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394112 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394116 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394119 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394123 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394127 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394130 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394133 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394136 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394139 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394142 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394144 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394147 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394150 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394153 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394158 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394161 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394163 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394168 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:38:33.403411 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394172 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394175 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394178 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394180 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394183 2580 flags.go:64] FLAG: --v="2" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394188 2580 flags.go:64] FLAG: --version="false" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394192 2580 flags.go:64] FLAG: --vmodule="" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394196 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394200 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394338 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394342 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394344 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394347 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394350 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394352 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394355 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394357 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394362 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394365 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394367 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394370 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.404672 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394372 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394375 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394377 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394380 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394382 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394385 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394389 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394393 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394396 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394398 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394401 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394405 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394407 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394410 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394413 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394415 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394418 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394420 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394423 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.405336 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394425 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394429 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394432 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394435 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394438 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394441 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394443 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394446 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394449 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394452 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394456 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394459 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394461 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394463 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394466 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394468 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394471 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394473 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394476 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.406113 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394479 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394481 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394484 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394486 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394506 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394511 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394514 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394517 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394520 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394522 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394525 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394527 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394530 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394532 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394535 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394538 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394540 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394543 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394546 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394548 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.406884 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394551 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394554 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394558 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394560 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394563 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394566 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394568 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394571 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394573 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394576 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394579 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394581 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394584 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394586 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394589 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394591 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394594 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394597 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394600 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394602 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.407522 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394605 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394607 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394609 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394612 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394615 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394617 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394620 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394622 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394625 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394627 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394630 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394632 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394635 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394637 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394641 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394643 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.408106 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.394685 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394722 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394727 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394730 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394733 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394735 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394738 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394740 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394743 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394746 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394748 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394751 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394753 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394756 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394759 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394762 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394764 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394767 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394769 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.408494 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394772 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394774 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394777 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394779 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394783 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394787 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394790 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394793 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394795 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394798 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394800 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394803 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394806 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394808 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394811 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394813 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394816 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394818 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394822 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.409043 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394826 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394828 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394831 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394834 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394837 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394840 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394843 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394845 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394848 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394850 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394853 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394855 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394858 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394860 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394863 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394865 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394867 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394870 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394873 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.409567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394875 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394878 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394880 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394883 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394885 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394888 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394890 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394892 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394895 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394897 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394900 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394903 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394905 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394908 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394911 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394913 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394916 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394918 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394921 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394923 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.410153 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394926 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394928 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394931 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394933 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394936 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394938 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394940 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394943 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394946 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394949 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394951 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394953 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394956 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394958 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394960 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394963 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394965 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394968 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394971 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394973 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.410959 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394975 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394978 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394981 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394984 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394987 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394989 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394991 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394994 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394996 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.394999 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.402609 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.402631 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.402649 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.402677 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402761 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402772 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402779 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402784 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402789 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402794 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.411878 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402798 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402803 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402807 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402811 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402815 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402819 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402823 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402827 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402831 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402836 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402840 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402844 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402848 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402852 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402857 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402861 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402865 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402869 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402873 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402877 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.412450 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402882 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402886 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402890 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402896 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402901 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402905 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402909 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402914 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402918 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402922 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402927 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402931 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402935 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402939 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402943 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402947 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402951 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402955 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402959 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.412978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402963 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402967 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402972 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402976 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402980 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402984 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402988 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402992 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.402996 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403000 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403005 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403009 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403013 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403017 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403022 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403025 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403030 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403035 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403040 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.413412 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403044 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403048 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403052 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403056 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403060 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403064 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403068 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403072 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403076 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403080 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403084 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403088 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403092 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403096 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403101 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403105 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403109 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403113 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403117 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.413905 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403121 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403127 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403133 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403139 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403144 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403148 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403153 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403157 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403161 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403165 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403170 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403175 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403181 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403185 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403189 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403198 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403202 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403206 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.414344 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403211 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403215 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403219 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403223 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.403265 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403500 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403509 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403514 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403518 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403522 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403527 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403531 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403536 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403540 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403544 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403548 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.415080 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403552 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403556 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403561 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403565 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403570 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403574 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403579 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403583 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403587 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403591 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403596 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403600 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403604 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403608 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403612 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403616 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403621 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403627 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403633 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403638 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.415487 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403642 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403646 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403650 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403673 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403677 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403680 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403684 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403688 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403692 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403696 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403699 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403703 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403707 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403710 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403714 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403718 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403723 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403726 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403730 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403734 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.416034 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403738 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403742 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403745 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403750 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403754 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403757 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403761 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403765 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403769 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403773 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403777 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403780 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403784 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403792 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403796 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403800 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403804 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403808 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403813 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.416542 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403817 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403820 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403825 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403829 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403833 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403838 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403843 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403847 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403851 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403855 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403859 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403864 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403868 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403873 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403877 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403881 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403885 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403890 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403894 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403899 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.417014 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403903 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403907 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403911 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403915 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403919 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403924 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403930 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403934 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403941 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403946 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403952 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403956 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403961 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403965 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403970 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.403974 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.417529 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.404020 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404082 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404089 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404093 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404098 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404102 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404106 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404110 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404115 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404119 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404123 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404127 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404132 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404136 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404141 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404145 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404149 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404153 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404158 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404163 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:33.417947 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404168 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404172 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404178 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404185 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404189 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404197 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404204 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404211 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404215 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404220 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404225 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404229 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404233 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404237 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404242 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404246 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404250 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404255 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:33.418579 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404259 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404264 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404268 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404272 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404276 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404281 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404286 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404290 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404295 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404300 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404304 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404308 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404312 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404316 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404320 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404327 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404332 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404336 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404341 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404345 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:33.419175 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404349 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404353 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404357 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404361 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404366 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404370 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404374 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404378 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404382 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404387 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404391 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404395 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404400 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404404 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404408 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404412 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404416 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404420 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404424 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:33.420017 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404429 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404433 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404437 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404441 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404445 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404449 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404454 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404458 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404463 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404469 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404473 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404477 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404482 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404486 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404490 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404494 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404498 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404503 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404507 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:33.420514 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404513 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404519 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404526 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404530 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404534 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404538 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404542 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404546 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404550 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:33.404554 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.404639 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.407106 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.408667 2580 server.go:1014] "Starting client certificate rotation" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.408771 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.409345 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.410592 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:33.421004 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.415036 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:33.428488 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.428458 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:38:33.431650 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.431630 2580 log.go:25] "Validated CRI v1 image API" Oct 30 20:38:33.432596 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.432568 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:38:33.434941 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.434919 2580 fs.go:135] Filesystem UUIDs: map[550086cb-f288-4101-bc81-e1c3af8f96f7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fce9c592-0ba1-40c5-a991-a65b31c4e745:/dev/nvme0n1p4] Oct 30 20:38:33.435001 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.434941 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:38:33.441065 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.440930 2580 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:33.439734585 +0000 UTC m=+0.305528801 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100067 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bcc768e77836fda87ad7dc98f998 SystemUUID:ec24bcc7-68e7-7836-fda8-7ad7dc98f998 BootID:6a7b7e2c-fd7f-4a19-9f27-118b7aced702 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:ab:86:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:ab:86:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:d9:c3:ac:86:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:38:33.441561 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.441550 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:38:33.441733 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.441719 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:38:33.444113 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444073 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:38:33.444262 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444114 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-65.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:38:33.444306 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444264 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:38:33.444306 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444273 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:38:33.444370 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444316 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:33.444370 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.444329 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:33.445091 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.445081 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:38:33.446686 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.446649 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:38:33.448428 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448412 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:38:33.448575 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448564 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:38:33.448634 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448607 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:38:33.448634 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448631 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:38:33.448749 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448646 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:38:33.448749 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448676 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:38:33.448749 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.448692 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:38:33.449084 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.449070 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:33.451158 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.451125 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:38:33.454913 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.454898 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:33.455528 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455515 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:38:33.455571 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455547 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:38:33.455744 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455733 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:38:33.455794 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455754 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:38:33.455794 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455780 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:38:33.455849 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455796 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:38:33.455849 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455801 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:38:33.455849 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.455807 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:38:33.456242 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456234 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:38:33.456272 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456242 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:38:33.456272 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456249 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:38:33.456272 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456256 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:38:33.456272 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456264 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:38:33.456272 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456273 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:38:33.456827 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456816 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:38:33.456862 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.456831 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:38:33.461813 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.461801 2580 server.go:1263] "Started kubelet" Oct 30 20:38:33.461912 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.461885 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:38:33.461980 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.461900 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:38:33.462028 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.461982 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:38:33.462533 ip-10-0-141-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:38:33.463064 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.463039 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:38:33.467139 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.467110 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:38:33.467806 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.467782 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-65.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:38:33.469840 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.469816 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:38:33.470807 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.470783 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7w88" Oct 30 20:38:33.471553 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.471532 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:38:33.471553 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.471542 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:33.471722 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.471706 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.472224 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.470923 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-65.us-east-2.compute.internal.18e3692bcca9965f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-65.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-65.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:38:33.461765727 +0000 UTC m=+0.327559944,LastTimestamp:2026-10-30 20:38:33.461765727 +0000 UTC m=+0.327559944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-65.us-east-2.compute.internal,}" Oct 30 20:38:33.472224 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472083 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:38:33.472224 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472110 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:38:33.472224 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472129 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:38:33.472459 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472237 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:38:33.472459 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472248 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:38:33.472459 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.472238 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:38:33.472459 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.472280 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-65.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:38:33.472822 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472771 2580 factory.go:153] Registering CRI-O factory Oct 30 20:38:33.472822 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472794 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 20:38:33.472964 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472851 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:38:33.472964 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472860 2580 factory.go:55] Registering systemd factory Oct 30 20:38:33.472964 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472865 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:38:33.472964 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472882 2580 factory.go:103] Registering Raw factory Oct 30 20:38:33.472964 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.472963 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 20:38:33.473950 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.473923 2580 manager.go:319] Starting recovery of all containers Oct 30 20:38:33.478108 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.478032 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7w88" Oct 30 20:38:33.484610 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.484573 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-65.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:38:33.485275 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.485253 2580 manager.go:324] Recovery completed Oct 30 20:38:33.488720 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.488701 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:38:33.490666 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.490640 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.492896 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.492879 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.492971 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.492911 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.492971 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.492924 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.493522 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.493508 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:38:33.493522 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.493519 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:38:33.493638 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.493535 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:38:33.493638 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.493556 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.494306 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.494284 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.494366 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.494320 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.494366 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.494333 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.495779 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.495765 2580 policy_none.go:50] "Start" Oct 30 20:38:33.495863 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.495782 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:38:33.495863 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.495796 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:38:33.497792 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.497782 2580 policy_none.go:44] "Start" Oct 30 20:38:33.535554 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.535354 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:38:33.539848 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.539830 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 20:38:33.539988 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.539861 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:38:33.539988 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.539870 2580 server.go:87] "Starting device plugin registration server" Oct 30 20:38:33.540143 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.540130 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:38:33.540222 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.540142 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:38:33.540274 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.540255 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:38:33.540370 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.540354 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:38:33.540370 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.540365 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:38:33.541032 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.541010 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:38:33.541129 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.541076 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.612254 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.612220 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:38:33.614246 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.613605 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:38:33.614246 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.613624 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:38:33.614246 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.613649 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:38:33.614246 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.613714 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:38:33.616550 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.616528 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:33.640844 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.640767 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.641870 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.641854 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.641935 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.641885 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.641935 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.641897 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.641935 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.641922 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.649716 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.649694 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.649830 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.649727 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-65.us-east-2.compute.internal\": node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.662492 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.662463 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.714521 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.714484 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal"] Oct 30 20:38:33.714706 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.714553 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.718536 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.718520 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.718615 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.718549 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.718615 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.718560 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.720862 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.720846 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.721015 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.721054 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721030 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.721724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721707 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.721812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721741 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.721812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721757 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.721812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721761 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.721812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721786 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.721812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.721800 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.723832 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.723817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.723886 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.723847 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:33.724591 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.724574 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:33.724591 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.724596 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:33.724785 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.724605 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:33.740148 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.740125 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.746505 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.746487 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.763059 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.763040 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.774572 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.774552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.774650 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.774577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.774650 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.774594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.863812 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.863776 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:33.875294 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.875454 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.875454 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.875454 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875403 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.875454 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875414 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.875603 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:33.875403 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:33.964505 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:33.964429 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.042959 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.042919 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:34.049647 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.049628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:34.065029 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:34.065003 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.165626 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:34.165584 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.266121 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:34.266058 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.366575 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:34.366534 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.409069 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.409027 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:38:34.409603 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.409224 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 20:38:34.467524 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:34.467489 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:38:34.487346 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.487308 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:33 +0000 UTC" deadline="2028-08-11 04:39:12.51769051 +0000 UTC" Oct 30 20:38:34.487346 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.487336 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15608h0m38.030357044s" Oct 30 20:38:34.504129 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.504100 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:34.510213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.510187 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:34.572212 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.572154 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:34.581741 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.581715 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:34.582819 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.582804 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:38:34.592996 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.592976 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:34.706230 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:34.706194 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999d3d59afc4db3f7e0da998760ec141.slice/crio-6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1 WatchSource:0}: Error finding container 6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1: Status 404 returned error can't find the container with id 6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1 Oct 30 20:38:34.712906 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.712890 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:38:34.716234 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:34.716213 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ada5c68d422440bbf7fd2285220de0.slice/crio-e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba WatchSource:0}: Error finding container e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba: Status 404 returned error can't find the container with id e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba Oct 30 20:38:34.725005 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:34.724986 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:35.450117 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.450029 2580 apiserver.go:52] "Watching apiserver" Oct 30 20:38:35.456339 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.456306 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:38:35.457152 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.457127 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-5cmmq","openshift-multus/multus-65qrq","openshift-network-operator/iptables-alerter-x874n","kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-gd9bl","openshift-multus/network-metrics-daemon-bdndq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz","openshift-cluster-node-tuning-operator/tuned-vbqx9","openshift-network-diagnostics/network-check-target-ktb85","kube-system/global-pull-secret-syncer-wnpsx","openshift-dns/node-resolver-bfbs4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-z55gr","kube-system/konnectivity-agent-ll9wf"] Oct 30 20:38:35.460152 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.460106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.460310 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.460205 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:35.462512 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.462490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.465805 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.465779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.467751 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.467720 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:38:35.467897 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.467838 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:38:35.467897 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.467856 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:38:35.468030 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468013 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:38:35.468086 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468036 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.468144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468113 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hh9qp\"" Oct 30 20:38:35.468198 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468165 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.468282 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.468387 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.468365 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.469367 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.469205 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:38:35.469367 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.469269 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gdbd\"" Oct 30 20:38:35.469367 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.469333 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.470494 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.470469 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.471080 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.471058 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:38:35.471177 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.471103 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:38:35.471177 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.471143 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4khcx\"" Oct 30 20:38:35.472716 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.472178 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:35.473721 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.473721 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.473211 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:35.473721 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473506 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:38:35.473721 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473653 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kszdw\"" Oct 30 20:38:35.473974 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473786 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:38:35.474029 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473989 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.474029 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.473990 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.474029 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.474010 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:38:35.476228 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.476207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:35.476321 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.476300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:35.478833 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.478313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.481436 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.481052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.482902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnm4\" (UniqueName: \"kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.482946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.482985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.483440 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gl9h\" (UniqueName: \"kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.483959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.484215 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx7w\" (UniqueName: \"kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.485034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.484324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdjh\" (UniqueName: \"kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.487716 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.487685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.489524 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.489502 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:38:35.490565 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.490541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.510634 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.510606 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lphtg" Oct 30 20:38:35.520162 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.520118 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lphtg" Oct 30 20:38:35.574103 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.574066 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:38:35.577074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577038 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577074 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577124 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577154 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577171 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577204 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:38:35.577243 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577219 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.577628 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577328 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w89q8\"" Oct 30 20:38:35.577628 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577402 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.577628 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577481 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.577628 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577572 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7wk5\"" Oct 30 20:38:35.577886 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577654 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:38:35.577886 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577786 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7chw8\"" Oct 30 20:38:35.577886 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577848 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b74p\"" Oct 30 20:38:35.578079 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.577945 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:38:35.578138 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.578100 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g4v6z\"" Oct 30 20:38:35.585249 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.585416 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.585416 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.585511 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585456 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.585511 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.585626 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.585626 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:35.585626 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.585781 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.585781 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.585781 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.585781 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.585781 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585882 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.585961 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kjl\" (UniqueName: \"kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.585987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.586144 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzx65\" (UniqueName: \"kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdjh\" (UniqueName: \"kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.586354 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnm4\" (UniqueName: \"kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9m8\" (UniqueName: \"kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.586606 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5fw5\" (UniqueName: \"kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfq4\" (UniqueName: \"kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586901 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.586985 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gl9h\" (UniqueName: \"kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.587213 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587442 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587459 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587496 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587532 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587553 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587525 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587598 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587684 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587703 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587769 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587873 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587880 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.588074 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.587993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588182 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588202 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588326 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588343 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588335 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588477 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.588678 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.589012 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588797 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.588811 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:38:36.088780319 +0000 UTC m=+2.954574529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588963 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.588974 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx7w\" (UniqueName: \"kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589171 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.589853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.589822 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.589899 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:36.089874735 +0000 UTC m=+2.955668944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.589995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.590027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.590665 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.590159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.596787 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.596714 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.597077 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.596870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.602221 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.602156 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdjh\" (UniqueName: \"kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.602493 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.602157 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gl9h\" (UniqueName: \"kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.602493 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.602313 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnm4\" (UniqueName: \"kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:35.606257 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.606180 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx7w\" (UniqueName: \"kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.624849 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.623450 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerStarted","Data":"6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1"} Oct 30 20:38:35.629075 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.628980 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"f7ada5c68d422440bbf7fd2285220de0","Type":"ContainerStarted","Data":"e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba"} Oct 30 20:38:35.690622 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fw5\" (UniqueName: \"kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.690622 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfq4\" (UniqueName: \"kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.690880 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.690983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.691181 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kjl\" (UniqueName: \"kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.691530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691636 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691680 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691777 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691887 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.692026 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691944 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.692401 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692141 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.691640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzx65\" (UniqueName: \"kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9m8\" (UniqueName: \"kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692726 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692747 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692768 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692854 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.692871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693076 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.693378 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693077 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693117 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693181 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693198 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693262 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693271 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693327 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693669 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.693851 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.694319 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694012 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694330 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694358 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694418 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694297 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694643 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.694724 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694702 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.695146 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.695146 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694773 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.695146 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.694866 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.696412 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.696054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.699172 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.699146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.703075 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.702973 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:35.703075 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.703013 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:35.703075 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.703033 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:35.703368 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:35.703116 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:38:36.203091269 +0000 UTC m=+3.068885476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:35.706109 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.706042 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fw5\" (UniqueName: \"kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.706899 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.706872 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kjl\" (UniqueName: \"kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.707029 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.706931 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfq4\" (UniqueName: \"kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.708581 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.708552 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzx65\" (UniqueName: \"kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.709030 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.709004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9m8\" (UniqueName: \"kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.797275 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.797230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:38:35.802225 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.801971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:35.808807 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:35.808765 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1565be2d_9663_42b1_bc86_72e4263fe7c3.slice/crio-0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24 WatchSource:0}: Error finding container 0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24: Status 404 returned error can't find the container with id 0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24 Oct 30 20:38:35.809864 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.809842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:38:35.812099 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.812076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:35.814559 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:35.814525 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744cc86_1e11_4932_894f_8610acdcff4e.slice/crio-29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757 WatchSource:0}: Error finding container 29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757: Status 404 returned error can't find the container with id 29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757 Oct 30 20:38:35.837484 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.837454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65qrq" Oct 30 20:38:35.840218 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.840068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:38:35.844420 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.844215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:38:35.847006 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.846981 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:38:35.852196 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:35.852165 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f5a5d4_d276_45f8_869e_a41dbd0d445b.slice/crio-d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556 WatchSource:0}: Error finding container d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556: Status 404 returned error can't find the container with id d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556 Oct 30 20:38:35.852278 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.852220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:38:35.866140 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:35.866029 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93073bfa_1168_4c11_a950_1f75458694d1.slice/crio-a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17 WatchSource:0}: Error finding container a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17: Status 404 returned error can't find the container with id a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17 Oct 30 20:38:35.868026 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:38:35.867977 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a8078d_c222_4943_af36_4285c0b450cd.slice/crio-c83e6863751312c283009d87f63f5244dc36ddfc078be0fe02b58a1a31fd6bc4 WatchSource:0}: Error finding container c83e6863751312c283009d87f63f5244dc36ddfc078be0fe02b58a1a31fd6bc4: Status 404 returned error can't find the container with id c83e6863751312c283009d87f63f5244dc36ddfc078be0fe02b58a1a31fd6bc4 Oct 30 20:38:35.957249 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:35.957153 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:36.098371 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.098332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:36.098596 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.098401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:36.098596 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.098527 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:36.098700 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.098614 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:37.098592079 +0000 UTC m=+3.964386298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:36.098775 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.098533 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:36.098874 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.098848 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:38:37.098823875 +0000 UTC m=+3.964618103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:36.300578 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.300481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:36.300763 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.300632 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:36.300763 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.300649 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:36.300763 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.300659 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:36.300763 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.300762 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:38:37.300740888 +0000 UTC m=+4.166535123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:36.523047 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.522977 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:35 +0000 UTC" deadline="2028-07-24 04:52:00.396649454 +0000 UTC" Oct 30 20:38:36.523047 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.523044 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15176h13m23.873609691s" Oct 30 20:38:36.614952 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.614866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:36.615120 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:36.615015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:36.639029 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.638753 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" event={"ID":"47a8078d-c222-4943-af36-4285c0b450cd","Type":"ContainerStarted","Data":"c83e6863751312c283009d87f63f5244dc36ddfc078be0fe02b58a1a31fd6bc4"} Oct 30 20:38:36.640953 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.640915 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfbs4" event={"ID":"0b22b136-e779-4470-b5ab-b894512f0294","Type":"ContainerStarted","Data":"539f653922078e65310cb77c26fc1fc81706df4575fb4ab1d8a6332a51c0743d"} Oct 30 20:38:36.642853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.642822 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ll9wf" event={"ID":"8bd09df0-d50b-4f00-863d-276efe60e695","Type":"ContainerStarted","Data":"8308c9586783f8ef39ec3b10e5dbf805a5ceb1cd3dd9ab5f209a181446e36c93"} Oct 30 20:38:36.657804 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.656043 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" event={"ID":"51cdc667-f01e-435d-8cf8-d7e805e0efd6","Type":"ContainerStarted","Data":"94028320deb5796dddac04dd4dcc42ffcdb698130df2af2e21f796ceff03383d"} Oct 30 20:38:36.669194 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.669007 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757"} Oct 30 20:38:36.672783 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.672677 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x874n" event={"ID":"1565be2d-9663-42b1-bc86-72e4263fe7c3","Type":"ContainerStarted","Data":"0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24"} Oct 30 20:38:36.684141 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.684098 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerStarted","Data":"a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17"} Oct 30 20:38:36.693657 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.693612 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cmmq" event={"ID":"54acb4d1-67b7-4ac8-9884-11af4f940626","Type":"ContainerStarted","Data":"6f7f4eb58565f909e35ec5593293e41a8f48e100fb49663c20e51849bc23e0e5"} Oct 30 20:38:36.717793 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:36.717749 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65qrq" event={"ID":"01f5a5d4-d276-45f8-869e-a41dbd0d445b","Type":"ContainerStarted","Data":"d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556"} Oct 30 20:38:37.110267 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.110016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:37.110433 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.110319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:37.110433 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.110420 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:37.110433 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.110418 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:37.110611 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.110482 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:38:39.110463844 +0000 UTC m=+5.976258071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:37.110611 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.110501 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:39.110491792 +0000 UTC m=+5.976286001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:37.312145 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.312104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:37.312360 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.312340 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:37.312439 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.312369 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:37.312439 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.312384 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:37.312538 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.312445 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:38:39.312425858 +0000 UTC m=+6.178220068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:37.523985 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.523885 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:35 +0000 UTC" deadline="2028-04-07 15:04:34.980675861 +0000 UTC" Oct 30 20:38:37.523985 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.523930 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12594h25m57.456750392s" Oct 30 20:38:37.632888 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.632828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:37.633081 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.633064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:37.633577 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.632858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:37.633894 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:37.633709 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:37.752798 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.751949 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"f7ada5c68d422440bbf7fd2285220de0","Type":"ContainerStarted","Data":"bde3b6280409d8b27340dece405c918aa545c2f4571664b8a92adc67647b16eb"} Oct 30 20:38:37.758691 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.758652 2580 generic.go:356] "Generic (PLEG): container finished" podID="999d3d59afc4db3f7e0da998760ec141" containerID="774789a8b22300550f51b3cad7afe8e11414d7e45c445212e918ec786355797b" exitCode=0 Oct 30 20:38:37.758886 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.758731 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerDied","Data":"774789a8b22300550f51b3cad7afe8e11414d7e45c445212e918ec786355797b"} Oct 30 20:38:37.769126 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:37.768176 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" podStartSLOduration=2.768157285 podStartE2EDuration="2.768157285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:37.767911601 +0000 UTC m=+4.633705854" watchObservedRunningTime="2026-10-30 20:38:37.768157285 +0000 UTC m=+4.633951516" Oct 30 20:38:38.614827 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:38.614784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:38.615279 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:38.614940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:38.769661 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:38.769625 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerStarted","Data":"8aed3c7ce6ebdd998009d0f1c81478a430b77647832416b79425dcf823548948"} Oct 30 20:38:39.132179 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.132051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:39.132179 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.132139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:39.132431 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.132284 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:39.132431 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.132355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.13233414 +0000 UTC m=+9.998128370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:39.132431 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.132428 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:39.132646 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.132462 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.132450326 +0000 UTC m=+9.998244535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:39.334918 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.334142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:39.334918 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.334381 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:39.334918 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.334410 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:39.334918 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.334425 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:39.334918 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.334492 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.334471332 +0000 UTC m=+10.200265563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:39.617812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.616989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:39.617812 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.617140 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:39.617812 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.617530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:39.617812 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:39.617622 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:39.777944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.777812 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="03a1cb271ea24c87d172c50a0999533360e918fe4cc0fddc2056338cd7b49ef7" exitCode=0 Oct 30 20:38:39.778512 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.778482 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"03a1cb271ea24c87d172c50a0999533360e918fe4cc0fddc2056338cd7b49ef7"} Oct 30 20:38:39.805072 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:39.805000 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" podStartSLOduration=4.804976773 podStartE2EDuration="4.804976773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:38.789482859 +0000 UTC m=+5.655277090" watchObservedRunningTime="2026-10-30 20:38:39.804976773 +0000 UTC m=+6.670771005" Oct 30 20:38:40.613998 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:40.613945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:40.614270 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:40.614147 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:40.788945 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:40.788119 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ll9wf" event={"ID":"8bd09df0-d50b-4f00-863d-276efe60e695","Type":"ContainerStarted","Data":"ae7a2053f50a8458eb15b56f5b058dfa9e969c2c4b8b53278993f5fbadb103d3"} Oct 30 20:38:40.806944 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:40.806609 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ll9wf" podStartSLOduration=2.5913837109999998 podStartE2EDuration="6.806583989s" totalImagesPullingTime="4.215200278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:40.805354376 +0000 UTC m=+7.671148618" watchObservedRunningTime="2026-10-30 20:38:40.806583989 +0000 UTC m=+7.672378225" Oct 30 20:38:41.051866 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.051771 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:41.052899 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.052619 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:38:41.614381 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.614309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:41.614531 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:41.614421 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:41.614771 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.614748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:41.614883 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:41.614862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:41.795382 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.795344 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cmmq" event={"ID":"54acb4d1-67b7-4ac8-9884-11af4f940626","Type":"ContainerStarted","Data":"0402fd9fe2f13d6b37fffd94d077bbd9079f357f15d05a9e456aee5cc743acbc"} Oct 30 20:38:41.810292 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:41.810191 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5cmmq" podStartSLOduration=2.52763765 podStartE2EDuration="7.810171774s" totalImagesPullingTime="5.282534124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:41.809524873 +0000 UTC m=+8.675319099" watchObservedRunningTime="2026-10-30 20:38:41.810171774 +0000 UTC m=+8.675966001" Oct 30 20:38:42.613828 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:42.613802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:42.614022 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:42.613906 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:42.801395 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:42.801358 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" event={"ID":"51cdc667-f01e-435d-8cf8-d7e805e0efd6","Type":"ContainerStarted","Data":"e55887c6a2a660faa728bec752a341dd2dc08ea770d94dc000123e47f8bee6d7"} Oct 30 20:38:42.801888 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:42.801466 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:43.176983 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.176942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:43.177168 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.177018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:43.177230 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.177165 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:43.177283 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.177231 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:38:51.177212728 +0000 UTC m=+18.043006949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:43.177672 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.177638 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:43.177750 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.177710 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:51.177693658 +0000 UTC m=+18.043487877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:43.378605 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.378567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:43.378821 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.378768 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:43.378821 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.378789 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:43.378821 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.378803 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:43.378975 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.378860 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:38:51.378845804 +0000 UTC m=+18.244640008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:43.614485 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.614408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:43.614641 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.614546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:43.615008 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.614969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:43.615103 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:43.615078 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:43.910236 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:43.910174 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:38:44.548916 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.548796 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:43.910205607Z","UUID":"42db2f7f-26ff-474f-8c23-378253454688","Handler":null,"Name":"","Endpoint":""} Oct 30 20:38:44.550681 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.550484 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:38:44.550681 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.550509 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:38:44.614910 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.614876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:44.615073 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:44.615010 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:44.809024 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.808989 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bfbs4" event={"ID":"0b22b136-e779-4470-b5ab-b894512f0294","Type":"ContainerStarted","Data":"41da9ece643a6baf4e0b2d75df8cf50f6fc2a95b0a64e97e4ae67db782dccd2d"} Oct 30 20:38:44.811306 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.811225 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" event={"ID":"51cdc667-f01e-435d-8cf8-d7e805e0efd6","Type":"ContainerStarted","Data":"ab10e2a4b961193ec52f2d54177f120bf70da8355183e5e1b653ab2042f1003a"} Oct 30 20:38:44.814906 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.814878 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="626e9370e1fac73f99f91bcbcf94222d5d8fa8835ba339c3c839383d7d368854" exitCode=0 Oct 30 20:38:44.815031 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.814966 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"626e9370e1fac73f99f91bcbcf94222d5d8fa8835ba339c3c839383d7d368854"} Oct 30 20:38:44.817832 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.817808 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65qrq" event={"ID":"01f5a5d4-d276-45f8-869e-a41dbd0d445b","Type":"ContainerStarted","Data":"aaa46449f73a58a41e0295d0cfb6d21db9192058f5b86f1ec5f3d784d75c85e7"} Oct 30 20:38:44.849622 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:44.849557 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bfbs4" podStartSLOduration=2.362884559 podStartE2EDuration="10.849537012s" totalImagesPullingTime="8.486652453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:44.825908152 +0000 UTC m=+11.691702378" watchObservedRunningTime="2026-10-30 20:38:44.849537012 +0000 UTC m=+11.715331240" Oct 30 20:38:45.617955 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.617736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:45.618368 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.617793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:45.618368 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:45.618064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:45.618368 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:45.618162 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:45.823848 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.823746 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" event={"ID":"51cdc667-f01e-435d-8cf8-d7e805e0efd6","Type":"ContainerStarted","Data":"dd4c46145473b1d742dfdf2268c1331b3cf491be8de094ece65437da723f51e2"} Oct 30 20:38:45.826146 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.826111 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x874n" event={"ID":"1565be2d-9663-42b1-bc86-72e4263fe7c3","Type":"ContainerStarted","Data":"62c1846a3a8ce07c4111bbcd131c069349ef2bc124ec79ecc126bb3d141ad3b2"} Oct 30 20:38:45.830479 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.830423 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="877966eacb4b5201adae3281b40e8093d5c0748ffd5d6449aa40d20647fb8472" exitCode=0 Oct 30 20:38:45.831324 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.831299 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"877966eacb4b5201adae3281b40e8093d5c0748ffd5d6449aa40d20647fb8472"} Oct 30 20:38:45.843460 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.843400 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-65qrq" podStartSLOduration=3.488299334 podStartE2EDuration="11.843387136s" totalImagesPullingTime="8.355087802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:44.867825238 +0000 UTC m=+11.733619467" watchObservedRunningTime="2026-10-30 20:38:45.843387136 +0000 UTC m=+12.709181362" Oct 30 20:38:45.865521 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:45.865463 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" podStartSLOduration=2.998095824 podStartE2EDuration="11.865446892s" totalImagesPullingTime="8.867351068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:45.842915643 +0000 UTC m=+12.708709870" watchObservedRunningTime="2026-10-30 20:38:45.865446892 +0000 UTC m=+12.731241118" Oct 30 20:38:46.614430 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.614392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:46.614622 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:46.614524 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:46.835231 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.835138 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" event={"ID":"47a8078d-c222-4943-af36-4285c0b450cd","Type":"ContainerStarted","Data":"b8b2d9a8ee3ebf9001bfc3bfb51a6edbaf3daf69a99392cdd42c5136640c5bef"} Oct 30 20:38:46.838246 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.838210 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="74e7140056faf87d4b876d5651eb807292bf74bc71d4c641186f6c15ad08d62b" exitCode=0 Oct 30 20:38:46.838414 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.838301 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"74e7140056faf87d4b876d5651eb807292bf74bc71d4c641186f6c15ad08d62b"} Oct 30 20:38:46.856551 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.856501 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" podStartSLOduration=2.351592426 podStartE2EDuration="12.856486162s" totalImagesPullingTime="10.504893736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:46.856324152 +0000 UTC m=+13.722118379" watchObservedRunningTime="2026-10-30 20:38:46.856486162 +0000 UTC m=+13.722280388" Oct 30 20:38:46.857099 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:46.857069 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x874n" podStartSLOduration=4.666562896 podStartE2EDuration="12.85705686s" totalImagesPullingTime="8.190493964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:45.881881317 +0000 UTC m=+12.747675548" watchObservedRunningTime="2026-10-30 20:38:46.85705686 +0000 UTC m=+13.722851085" Oct 30 20:38:47.614998 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:47.614964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:47.615170 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:47.615093 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:47.615234 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:47.615192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:47.615354 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:47.615331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:48.614892 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:48.614864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:48.615405 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:48.614966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:48.845949 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:48.845863 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="29da43e242d395f60a93e802bf4fd9c212197b0bbb480e28e3b62bc71aa200a0" exitCode=0 Oct 30 20:38:48.845949 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:48.845931 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"29da43e242d395f60a93e802bf4fd9c212197b0bbb480e28e3b62bc71aa200a0"} Oct 30 20:38:49.618020 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:49.617803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:49.618450 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:49.617803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:49.618450 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:49.618097 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:49.618450 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:49.618176 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:49.851367 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:49.851331 2580 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="42b146a08d4d1cf4bea73cfb408f31b66f3c7d9b9a8b2147590c47b93db41851" exitCode=0 Oct 30 20:38:49.851523 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:49.851387 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerDied","Data":"42b146a08d4d1cf4bea73cfb408f31b66f3c7d9b9a8b2147590c47b93db41851"} Oct 30 20:38:50.614247 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:50.614208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:50.614534 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:50.614314 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:50.857739 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:50.857706 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" event={"ID":"93073bfa-1168-4c11-a950-1f75458694d1","Type":"ContainerStarted","Data":"8b1f76d285d288a1acd9b65ae6eb1285ccc18266156a7f0786b037e6635bc98b"} Oct 30 20:38:50.883840 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:50.883790 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gd9bl" podStartSLOduration=6.407937989 podStartE2EDuration="16.883777318s" totalImagesPullingTime="10.475839329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:50.883155982 +0000 UTC m=+17.748950208" watchObservedRunningTime="2026-10-30 20:38:50.883777318 +0000 UTC m=+17.749571543" Oct 30 20:38:51.243758 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.243721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:51.243958 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.243774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:51.243958 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.243894 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:51.243958 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.243893 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:51.244082 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.243967 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:07.243947154 +0000 UTC m=+34.109741358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:51.244082 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.243992 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:39:07.243977665 +0000 UTC m=+34.109771874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:51.445634 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.445389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:51.445634 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.445548 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:51.445899 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.445646 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:51.445899 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.445686 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:51.445899 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.445748 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:39:07.445731505 +0000 UTC m=+34.311525726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:51.617197 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.617160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:51.617197 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.617195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:51.617481 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.617288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:51.617481 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:51.617426 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:51.862535 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862445 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"5c0e3500a844f82a935a55f52b4a2159a32c76d48ceed3db79f9e2f5f2563e5b"} Oct 30 20:38:51.862937 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862536 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"beeb9ead9bae812aa16fbf2527cb2f512517c81a575456c5076eb6d641ba5510"} Oct 30 20:38:51.862937 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862586 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"b1d04eb33cdb171cbd1f7bfe1500fe3d0ff0bc370846181cab24ab8a8b78698f"} Oct 30 20:38:51.862937 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862633 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"94de81db38f9def3579d3b179233b43cb971aae0c4e19b8070cfd9826366e6e1"} Oct 30 20:38:51.862937 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862695 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"9d6e6678e35beb338ba5f8e008cf1987c74d65eb7b14e0f24f63781b65230513"} Oct 30 20:38:51.862937 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:51.862741 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"c7f113371dc2d2c24d55073005a8b8fe5cf8c2fd1129f04a7d3ebcfb9a6ba393"} Oct 30 20:38:52.614777 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:52.614749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:52.614937 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:52.614839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:53.617152 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:53.617074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:53.617553 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:53.617172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:53.617553 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:53.617224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:53.617553 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:53.617309 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:53.872767 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:53.872684 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"bb091b5ccc30fe427a8f9cdf4aa0eb0dbd877a9f4b83cef7a902e1a190a78218"} Oct 30 20:38:54.614057 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:54.614005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:54.614243 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:54.614120 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:55.618034 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.617942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:55.618810 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:55.618081 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:55.618810 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.618135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:55.618810 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:55.618253 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:55.880208 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.880108 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" event={"ID":"f744cc86-1e11-4932-894f-8610acdcff4e","Type":"ContainerStarted","Data":"47cddd7fe3c3b74138a71a834e6c3b87757a925d180bf1a0937148d4b77a6eac"} Oct 30 20:38:55.880455 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.880400 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:55.880495 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.880482 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:55.896545 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.896514 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:55.908312 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.908291 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bfbs4_0b22b136-e779-4470-b5ab-b894512f0294/dns-node-resolver/0.log" Oct 30 20:38:55.917949 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:55.917905 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" podStartSLOduration=6.8973731990000005 podStartE2EDuration="21.917891926s" totalImagesPullingTime="15.020518727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:55.91752812 +0000 UTC m=+22.783322347" watchObservedRunningTime="2026-10-30 20:38:55.917891926 +0000 UTC m=+22.783686151" Oct 30 20:38:56.614093 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:56.614057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:56.614242 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:56.614163 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:56.882330 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:56.882251 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:56.883979 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:56.883953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cmmq_54acb4d1-67b7-4ac8-9884-11af4f940626/node-ca/0.log" Oct 30 20:38:56.898555 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:56.898528 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:38:57.380022 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.379736 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wnpsx"] Oct 30 20:38:57.380168 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.380058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:57.380168 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:57.380148 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:57.383458 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.383429 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdndq"] Oct 30 20:38:57.383617 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.383538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:57.383687 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:57.383630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:38:57.384042 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.384022 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ktb85"] Oct 30 20:38:57.384150 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:57.384121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:57.384275 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:57.384228 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:59.614598 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:59.614564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:38:59.615015 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:59.614580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:38:59.615015 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:38:59.614693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:38:59.615015 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:59.614696 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:38:59.615015 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:59.614760 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:38:59.615015 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:38:59.614839 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:01.614733 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:01.614700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:01.614733 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:01.614737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:01.615195 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:01.614705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:01.615195 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:01.614828 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:01.615195 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:01.614883 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:01.615195 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:01.614956 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:03.616028 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:03.615964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:03.616028 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:03.616007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:03.616739 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:03.616153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:03.616739 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:03.616201 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:03.616739 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:03.616243 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:03.616739 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:03.616325 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:04.461473 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:04.461440 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:04.461647 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:04.461568 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:39:04.462143 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:04.462119 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:05.614410 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:05.614368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:05.614832 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:05.614368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:05.614832 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:05.614508 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:05.614832 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:05.614373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:05.614832 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:05.614584 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:05.614832 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:05.614630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:07.257155 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.257127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:07.257569 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.257170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:07.257569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.257268 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:39:07.257569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.257322 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:39:07.257569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.257337 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs podName:7bb3812a-4ba1-46e5-9433-c9f830ac9629 nodeName:}" failed. No retries permitted until 2026-10-30 20:39:39.257318755 +0000 UTC m=+66.123112974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs") pod "network-metrics-daemon-bdndq" (UID: "7bb3812a-4ba1-46e5-9433-c9f830ac9629") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:39:07.257569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.257356 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret podName:fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b nodeName:}" failed. No retries permitted until 2026-10-30 20:39:39.257345134 +0000 UTC m=+66.123139338 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret") pod "global-pull-secret-syncer-wnpsx" (UID: "fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:39:07.459372 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.459335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:07.459530 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.459495 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:39:07.459530 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.459510 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:39:07.459530 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.459520 2580 projected.go:196] Error preparing data for projected volume kube-api-access-s8vnf for pod openshift-network-diagnostics/network-check-target-ktb85: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:39:07.459636 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.459568 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf podName:a557f314-88c8-4807-aeb5-6e73cf74a44c nodeName:}" failed. No retries permitted until 2026-10-30 20:39:39.459555996 +0000 UTC m=+66.325350200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8vnf" (UniqueName: "kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf") pod "network-check-target-ktb85" (UID: "a557f314-88c8-4807-aeb5-6e73cf74a44c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:39:07.614899 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.614872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:07.614899 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.614887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:07.615067 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:07.614904 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:07.615067 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.614978 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:07.615067 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.615054 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:07.615172 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:07.615130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:09.614065 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:09.614040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:09.614469 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:09.614038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:09.614469 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:09.614138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:09.614469 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:09.614217 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:09.614469 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:09.614254 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:09.614469 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:09.614303 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:11.614097 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:11.614055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:11.614097 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:11.614085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:11.614569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:11.614161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:11.614569 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:11.614243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:11.614569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:11.614360 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:11.614569 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:11.614443 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:13.615171 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:13.615126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:13.615678 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:13.615221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:13.615678 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:13.615268 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:13.615678 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:13.615296 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:13.615678 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:13.615322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:13.615678 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:13.615396 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:15.614114 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:15.614066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:15.614114 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:15.614103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:15.614114 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:15.614118 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:15.614594 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:15.614229 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:15.614594 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:15.614332 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:15.614594 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:15.614429 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:17.614833 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:17.614798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:17.615295 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:17.614797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:17.615295 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:17.614912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:17.615295 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:17.614917 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:17.615295 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:17.615012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:17.615295 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:17.615100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:19.614617 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:19.614582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:19.615071 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:19.614593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:19.615071 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:19.614717 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:19.615071 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:19.614593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:19.615071 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:19.614771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:19.615071 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:19.614873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:21.614871 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:21.614836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:21.614871 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:21.614859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:21.615283 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:21.614937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:21.615283 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:21.614941 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:21.615283 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:21.615011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:21.615283 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:21.615084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:23.615408 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:23.615371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:23.615929 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:23.615471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdndq" podUID="7bb3812a-4ba1-46e5-9433-c9f830ac9629" Oct 30 20:39:23.615929 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:23.615550 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:23.615929 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:23.615641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:23.615929 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:23.615681 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wnpsx" podUID="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" Oct 30 20:39:23.615929 ip-10-0-141-65 kubenswrapper[2580]: E1030 20:39:23.615707 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ktb85" podUID="a557f314-88c8-4807-aeb5-6e73cf74a44c" Oct 30 20:39:24.926685 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.926596 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeReady" Oct 30 20:39:24.927049 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.926741 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:39:24.976936 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.976858 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m74jp"] Oct 30 20:39:24.991672 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.991626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cmmq_54acb4d1-67b7-4ac8-9884-11af4f940626/node-ca/0.log" Oct 30 20:39:24.998842 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.998804 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jggzv"] Oct 30 20:39:24.999017 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:24.998946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.007715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.007693 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:39:25.007981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.007965 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl5jd\"" Oct 30 20:39:25.008067 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.008049 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:39:25.014751 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.014725 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m74jp"] Oct 30 20:39:25.014876 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.014767 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jggzv"] Oct 30 20:39:25.014876 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.014784 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wm9wk"] Oct 30 20:39:25.014876 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.014837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.028945 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.028921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.029501 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.029477 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wm9wk"] Oct 30 20:39:25.072530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.072530 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcqv\" (UniqueName: \"kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.072803 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.072803 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.072803 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.072803 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.072803 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.073039 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf98h\" (UniqueName: \"kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.073039 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vcm\" (UniqueName: \"kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.073039 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.073039 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.072926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.115064 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.115032 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:39:25.116331 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116315 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:39:25.116401 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116358 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2twjb\"" Oct 30 20:39:25.116401 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116360 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:39:25.116401 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116376 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:39:25.116532 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116454 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:39:25.116532 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116468 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:39:25.116532 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116489 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:39:25.116532 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.116500 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cr4sq\"" Oct 30 20:39:25.174087 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.174185 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcqv\" (UniqueName: \"kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174249 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174302 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.174352 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.174352 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.174439 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174439 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.174539 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf98h\" (UniqueName: \"kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.174539 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vcm\" (UniqueName: \"kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.174633 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174582 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174633 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174621 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174932 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.174932 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.174690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.175053 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.175027 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.175130 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.175112 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.178158 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.178087 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.178158 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.178095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.178307 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.178248 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.186175 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.186151 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vcm\" (UniqueName: \"kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.186280 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.186201 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcqv\" (UniqueName: \"kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.186280 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.186240 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf98h\" (UniqueName: \"kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.336085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.336042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:25.356232 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.356201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:25.399348 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.399323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:25.614853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.614812 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:25.614853 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.614844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:25.615082 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.614858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618335 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ndwvd\"" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618610 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618764 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5p7gr\"" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618823 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618874 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:39:25.618981 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.618926 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5kr7\"" Oct 30 20:39:25.619318 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.619049 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:39:25.689505 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.689429 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jggzv"] Oct 30 20:39:25.692303 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.692258 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m74jp"] Oct 30 20:39:25.693466 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:25.693431 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8866b225_b24b_43f3_b2a5_960fa1d9f751.slice/crio-8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383 WatchSource:0}: Error finding container 8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383: Status 404 returned error can't find the container with id 8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383 Oct 30 20:39:25.695313 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:25.695289 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd103ef_40a7_49a8_a4c7_bc5295ff7cc4.slice/crio-a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734 WatchSource:0}: Error finding container a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734: Status 404 returned error can't find the container with id a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734 Oct 30 20:39:25.731062 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.731031 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wm9wk"] Oct 30 20:39:25.734392 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:25.734367 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3960ed57_c28a_413f_817e_355fa6029130.slice/crio-1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4 WatchSource:0}: Error finding container 1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4: Status 404 returned error can't find the container with id 1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4 Oct 30 20:39:25.945701 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.945598 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wm9wk" event={"ID":"3960ed57-c28a-413f-817e-355fa6029130","Type":"ContainerStarted","Data":"377f399481679c7e77476c0fdc854ef091b81c2a67bdebed80c5ac3aa3f9a6ac"} Oct 30 20:39:25.945701 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.945689 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wm9wk" event={"ID":"3960ed57-c28a-413f-817e-355fa6029130","Type":"ContainerStarted","Data":"1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4"} Oct 30 20:39:25.946879 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.946854 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jggzv" event={"ID":"8866b225-b24b-43f3-b2a5-960fa1d9f751","Type":"ContainerStarted","Data":"8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383"} Oct 30 20:39:25.948059 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:25.948034 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m74jp" event={"ID":"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4","Type":"ContainerStarted","Data":"a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734"} Oct 30 20:39:26.093861 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.093824 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz"] Oct 30 20:39:26.117897 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.117860 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz"] Oct 30 20:39:26.117897 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.117876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.121118 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121093 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:39:26.121285 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:39:26.121365 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121353 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 20:39:26.121421 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121407 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:39:26.121457 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121434 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hpn29\"" Oct 30 20:39:26.121510 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.121491 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 20:39:26.183316 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.183281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.183478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.183337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.183478 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.183423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.183617 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.183480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rng4\" (UniqueName: \"kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.283986 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.283899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.284148 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.283983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.284148 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.284023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.284148 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.284082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rng4\" (UniqueName: \"kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.285492 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.285412 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.287904 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.287846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.288228 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.287927 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.300188 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.300131 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rng4\" (UniqueName: \"kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.454260 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.454223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:26.765858 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.765827 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz"] Oct 30 20:39:26.769567 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:26.769536 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eca690e_c8c5_4526_bdc4_7cb0a436fc30.slice/crio-22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7 WatchSource:0}: Error finding container 22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7: Status 404 returned error can't find the container with id 22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7 Oct 30 20:39:26.951922 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:26.951881 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" event={"ID":"4eca690e-c8c5-4526-bdc4-7cb0a436fc30","Type":"ContainerStarted","Data":"22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7"} Oct 30 20:39:27.956879 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:27.956836 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wm9wk" event={"ID":"3960ed57-c28a-413f-817e-355fa6029130","Type":"ContainerStarted","Data":"352ef11ef42c77a3896801a32bc3400f774a52dac493ec24b5256f6b76478e06"} Oct 30 20:39:28.906318 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.905216 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:28.962725 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.962683 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" event={"ID":"4eca690e-c8c5-4526-bdc4-7cb0a436fc30","Type":"ContainerStarted","Data":"bbaeeacea5d947fede34131087400e3777dcecedaeeb55b244f03bbf7ddef8a1"} Oct 30 20:39:28.963192 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.962756 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" event={"ID":"4eca690e-c8c5-4526-bdc4-7cb0a436fc30","Type":"ContainerStarted","Data":"3c079bd40005839468dd6fc975d66553d0885abef8e97f8af48b969911f0cdbf"} Oct 30 20:39:28.964585 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.964559 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m74jp" event={"ID":"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4","Type":"ContainerStarted","Data":"56eda0ed547b2dadb732e2d50829fee38a00fd6e7dfe0801bdbc5a941f257656"} Oct 30 20:39:28.964746 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.964608 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m74jp" event={"ID":"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4","Type":"ContainerStarted","Data":"0bed50878f2b38e406f5c4c9bf63d909246165b8d6144cc365c1a5013bdff4a3"} Oct 30 20:39:28.964746 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.964696 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:28.966349 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.966317 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jggzv" event={"ID":"8866b225-b24b-43f3-b2a5-960fa1d9f751","Type":"ContainerStarted","Data":"a3642877c471d488d3ebb8fb9fd88ce41ff53251654b7785514de619eb179e4e"} Oct 30 20:39:28.968371 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.968349 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wm9wk" event={"ID":"3960ed57-c28a-413f-817e-355fa6029130","Type":"ContainerStarted","Data":"fde277c45d00b1af9466f2876a0dccd5dfbdbe7855f7a4ea08527b5c0d5553d9"} Oct 30 20:39:28.981224 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:28.981170 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" podStartSLOduration=1.231197211 podStartE2EDuration="2.981154146s" totalImagesPullingTime="1.749956935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:28.980903719 +0000 UTC m=+55.846698051" watchObservedRunningTime="2026-10-30 20:39:28.981154146 +0000 UTC m=+55.846948371" Oct 30 20:39:29.018857 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:29.018810 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-m74jp" podStartSLOduration=2.724178599 podStartE2EDuration="5.018795678s" totalImagesPullingTime="2.294617079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:29.001885951 +0000 UTC m=+55.867680198" watchObservedRunningTime="2026-10-30 20:39:29.018795678 +0000 UTC m=+55.884589949" Oct 30 20:39:29.019085 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:29.019067 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jggzv" podStartSLOduration=1.924424003 podStartE2EDuration="5.01906004s" totalImagesPullingTime="3.094636037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:29.018137339 +0000 UTC m=+55.883931583" watchObservedRunningTime="2026-10-30 20:39:29.01906004 +0000 UTC m=+55.884854266" Oct 30 20:39:29.039877 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:29.038882 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wm9wk" podStartSLOduration=2.1723951599999998 podStartE2EDuration="5.038863443s" totalImagesPullingTime="2.866468283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:29.038494153 +0000 UTC m=+55.904288406" watchObservedRunningTime="2026-10-30 20:39:29.038863443 +0000 UTC m=+55.904657672" Oct 30 20:39:30.477120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.477089 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xx98g"] Oct 30 20:39:30.480297 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.480279 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb"] Oct 30 20:39:30.480429 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.480410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.483171 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.483149 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8"] Oct 30 20:39:30.483314 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.483266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.486433 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.486407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.494938 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.494860 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:39:30.494938 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.494890 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 20:39:30.494938 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.494875 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495635 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dctv6\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495636 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495755 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495781 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9jnnb\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495833 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6ml5f\"" Oct 30 20:39:30.495914 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495892 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 20:39:30.496162 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.495927 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:39:30.496277 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.496259 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:39:30.496360 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.496300 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8"] Oct 30 20:39:30.497371 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.497304 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb"] Oct 30 20:39:30.521700 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.521884 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.521884 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.521884 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.521884 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.521884 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.522120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.522120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.521964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.522120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4thr\" (UniqueName: \"kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.522120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.522120 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km96v\" (UniqueName: \"kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.522311 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.522514 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.522514 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.522389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.623472 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.623472 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623562 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4thr\" (UniqueName: \"kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.623768 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.624186 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624164 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.624261 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624238 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624305 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.623862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624387 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624423 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624387 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.624465 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.624465 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km96v\" (UniqueName: \"kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.624546 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624546 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624469 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.624546 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.624546 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624526 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624764 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624764 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.624764 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624764 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.624968 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.624968 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.624824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.625458 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.625143 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.625458 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.625187 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.625675 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.625628 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.626158 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.626009 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.627052 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627028 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.627170 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.627788 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.627788 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627766 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.627949 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.627999 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.627955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.636439 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.636411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km96v\" (UniqueName: \"kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.636574 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.636407 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4thr\" (UniqueName: \"kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.636574 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.636496 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.798523 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.798494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:30.805666 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.805608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:30.811842 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.811807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:30.812207 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:30.812181 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f2b5e0_aaed_456d_a023_c2bbf64765dd.slice/crio-845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57 WatchSource:0}: Error finding container 845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57: Status 404 returned error can't find the container with id 845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57 Oct 30 20:39:30.980855 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:30.980815 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx98g" event={"ID":"16f2b5e0-aaed-456d-a023-c2bbf64765dd","Type":"ContainerStarted","Data":"845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57"} Oct 30 20:39:31.086979 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.086881 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb"] Oct 30 20:39:31.089750 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:31.089720 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26071f56_69e8_4327_a05f_deaf1f8e209a.slice/crio-a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330 WatchSource:0}: Error finding container a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330: Status 404 returned error can't find the container with id a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330 Oct 30 20:39:31.095334 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.095307 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8"] Oct 30 20:39:31.097908 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:31.097885 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0208ffae_7118_4175_8e46_97ac0592ebfc.slice/crio-18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006 WatchSource:0}: Error finding container 18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006: Status 404 returned error can't find the container with id 18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006 Oct 30 20:39:31.987125 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.987093 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx98g" event={"ID":"16f2b5e0-aaed-456d-a023-c2bbf64765dd","Type":"ContainerStarted","Data":"d836cd9fef8d0481ab02d867a3a8f15d6a986b99b4682cd188ea949fb6104ac5"} Oct 30 20:39:31.990241 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.990183 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" event={"ID":"0208ffae-7118-4175-8e46-97ac0592ebfc","Type":"ContainerStarted","Data":"bcc36d1638e3ef47ee8df32c3f99d3a0647ae6c46a8e8f24876c942b60c23577"} Oct 30 20:39:31.990383 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.990255 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" event={"ID":"0208ffae-7118-4175-8e46-97ac0592ebfc","Type":"ContainerStarted","Data":"922abf275d687948df197bbc00f3bde24aa6b1ffcbdd2d98d356a8286a97139f"} Oct 30 20:39:31.990383 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.990284 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" event={"ID":"0208ffae-7118-4175-8e46-97ac0592ebfc","Type":"ContainerStarted","Data":"18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006"} Oct 30 20:39:31.993125 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:31.993099 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" event={"ID":"26071f56-69e8-4327-a05f-deaf1f8e209a","Type":"ContainerStarted","Data":"a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330"} Oct 30 20:39:32.997707 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:32.997650 2580 generic.go:356] "Generic (PLEG): container finished" podID="16f2b5e0-aaed-456d-a023-c2bbf64765dd" containerID="d836cd9fef8d0481ab02d867a3a8f15d6a986b99b4682cd188ea949fb6104ac5" exitCode=0 Oct 30 20:39:32.998202 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:32.997754 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx98g" event={"ID":"16f2b5e0-aaed-456d-a023-c2bbf64765dd","Type":"ContainerDied","Data":"d836cd9fef8d0481ab02d867a3a8f15d6a986b99b4682cd188ea949fb6104ac5"} Oct 30 20:39:33.000260 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.000155 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" event={"ID":"0208ffae-7118-4175-8e46-97ac0592ebfc","Type":"ContainerStarted","Data":"d272d74eb6d6cc1883e71f085b75309c1cb08d2afe1b500f05e57fdb2249b8ba"} Oct 30 20:39:33.002307 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.002283 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" event={"ID":"26071f56-69e8-4327-a05f-deaf1f8e209a","Type":"ContainerStarted","Data":"528632328c67042d8157a646f05b0a020536fb75ea22cf15fb7773ae2c0acdff"} Oct 30 20:39:33.002441 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.002346 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" event={"ID":"26071f56-69e8-4327-a05f-deaf1f8e209a","Type":"ContainerStarted","Data":"8aa93b122cff9f7ad979f08cd08d95c8d4d3c70e83e160c5de113a7a053b9897"} Oct 30 20:39:33.002441 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.002375 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" event={"ID":"26071f56-69e8-4327-a05f-deaf1f8e209a","Type":"ContainerStarted","Data":"d2b49773d960058277c42dc6a0db274d0c78a2aa3677170776fba9c5146321f3"} Oct 30 20:39:33.044991 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.044943 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" podStartSLOduration=2.041181861 podStartE2EDuration="3.044926953s" totalImagesPullingTime="1.003745092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:33.043495918 +0000 UTC m=+59.909290145" watchObservedRunningTime="2026-10-30 20:39:33.044926953 +0000 UTC m=+59.910721189" Oct 30 20:39:33.065309 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:33.065174 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" podStartSLOduration=1.807882861 podStartE2EDuration="3.065159701s" totalImagesPullingTime="1.25727684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:33.063755669 +0000 UTC m=+59.929549895" watchObservedRunningTime="2026-10-30 20:39:33.065159701 +0000 UTC m=+59.930953926" Oct 30 20:39:34.011823 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.011785 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx98g" event={"ID":"16f2b5e0-aaed-456d-a023-c2bbf64765dd","Type":"ContainerStarted","Data":"6b69a79a80e9a899d16e2693a0598262b20cd0862293eea27b59a518e86d9426"} Oct 30 20:39:34.012324 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.011866 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xx98g" event={"ID":"16f2b5e0-aaed-456d-a023-c2bbf64765dd","Type":"ContainerStarted","Data":"9d55b0e42a6bd299896338a88bff368f7ee4144d42480f4763da7e2ca121e54f"} Oct 30 20:39:34.033926 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.033865 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xx98g" podStartSLOduration=2.873521361 podStartE2EDuration="4.033848689s" totalImagesPullingTime="1.068632084s" totalInitContainerRuntime="91.695244ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:34.032697621 +0000 UTC m=+60.898491892" watchObservedRunningTime="2026-10-30 20:39:34.033848689 +0000 UTC m=+60.899642915" Oct 30 20:39:34.850403 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.850373 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-656d8b8c49-pmmgw"] Oct 30 20:39:34.854850 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.854830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.862574 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.862549 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 20:39:34.862764 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.862602 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:39:34.864095 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.864041 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-656d8b8c49-pmmgw"] Oct 30 20:39:34.886360 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.886326 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b2k4x\"" Oct 30 20:39:34.886360 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.886336 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 20:39:34.886360 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.886368 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 20:39:34.886727 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.886402 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a0bik8i2clpn0\"" Oct 30 20:39:34.966250 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966250 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbt9\" (UniqueName: \"kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966438 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966475 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966507 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966507 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:34.966565 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:34.966526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067232 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.067995 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.067725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbt9\" (UniqueName: \"kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.068227 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.068195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.068334 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.068247 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.068334 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.068315 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.070255 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.070236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.070711 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.070692 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.070795 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.070730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.081351 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.081314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbt9\" (UniqueName: \"kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.166367 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.166271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:35.234845 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.234813 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dck4k"] Oct 30 20:39:35.240065 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.239479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.243876 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.243858 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 20:39:35.244250 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.244233 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qc5gl\"" Oct 30 20:39:35.255048 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.254993 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dck4k"] Oct 30 20:39:35.370523 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.370478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.370711 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.370542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqpw\" (UniqueName: \"kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.445788 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.445711 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-656d8b8c49-pmmgw"] Oct 30 20:39:35.448976 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:35.448949 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac2e542_da70_4445_a126_d02866a7e2f9.slice/crio-e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75 WatchSource:0}: Error finding container e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75: Status 404 returned error can't find the container with id e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75 Oct 30 20:39:35.471582 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.471540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.471823 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.471596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqpw\" (UniqueName: \"kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.474231 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.474211 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.484053 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.484022 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqpw\" (UniqueName: \"kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.552950 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.552909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:35.842459 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:35.842425 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-dck4k"] Oct 30 20:39:35.844757 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:35.844727 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1912739c_4112_436d_9000_e5858dce9aeb.slice/crio-84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db WatchSource:0}: Error finding container 84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db: Status 404 returned error can't find the container with id 84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db Oct 30 20:39:36.019893 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.019844 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" event={"ID":"3ac2e542-da70-4445-a126-d02866a7e2f9","Type":"ContainerStarted","Data":"e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75"} Oct 30 20:39:36.021156 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.021126 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" event={"ID":"1912739c-4112-436d-9000-e5858dce9aeb","Type":"ContainerStarted","Data":"84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db"} Oct 30 20:39:36.587549 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.587516 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzv86"] Oct 30 20:39:36.590757 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.590734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.595298 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.595261 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:39:36.596193 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.595954 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:39:36.596326 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.596294 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-tcxgx\"" Oct 30 20:39:36.604852 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.602475 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzv86"] Oct 30 20:39:36.682468 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.682427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.682715 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.682498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnht\" (UniqueName: \"kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.783447 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.783407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.783648 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.783473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnht\" (UniqueName: \"kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.784261 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.784234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.805738 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.805704 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnht\" (UniqueName: \"kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:36.911596 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:36.911498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:37.039689 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:37.037988 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" event={"ID":"3ac2e542-da70-4445-a126-d02866a7e2f9","Type":"ContainerStarted","Data":"73dde30c7fb41446f177646d4e51219d7606617989fc2116312b47bbab013836"} Oct 30 20:39:37.061489 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:37.061437 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" podStartSLOduration=1.5887707519999998 podStartE2EDuration="3.061418733s" totalImagesPullingTime="1.472647981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:37.059413307 +0000 UTC m=+63.925207570" watchObservedRunningTime="2026-10-30 20:39:37.061418733 +0000 UTC m=+63.927212958" Oct 30 20:39:37.215018 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:37.214924 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-gzv86"] Oct 30 20:39:37.223795 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:37.223750 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3039da16_1889_4c78_b2c0_4b612a0cb125.slice/crio-4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058 WatchSource:0}: Error finding container 4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058: Status 404 returned error can't find the container with id 4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058 Oct 30 20:39:38.043759 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.043720 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" event={"ID":"1912739c-4112-436d-9000-e5858dce9aeb","Type":"ContainerStarted","Data":"9f437ac54313c8b7013c749c6485b37923b7efb2151488afe9c4fb92f7f02fe9"} Oct 30 20:39:38.044227 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.043942 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:38.045395 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.045365 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gzv86" event={"ID":"3039da16-1889-4c78-b2c0-4b612a0cb125","Type":"ContainerStarted","Data":"4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058"} Oct 30 20:39:38.050165 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.050140 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:38.060452 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.060397 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" podStartSLOduration=1.631072181 podStartE2EDuration="3.060381305s" totalImagesPullingTime="1.429309124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:38.060095053 +0000 UTC m=+64.925889280" watchObservedRunningTime="2026-10-30 20:39:38.060381305 +0000 UTC m=+64.926175532" Oct 30 20:39:38.977250 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:38.977216 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:39.311070 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.311028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:39.311528 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.311083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:39.313449 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.313424 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:39:39.314130 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.314111 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:39:39.324973 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.324944 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:39.324973 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.324965 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:39.462922 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.462882 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ndwvd\"" Oct 30 20:39:39.471078 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.471043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:39.506883 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.506848 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5p7gr\"" Oct 30 20:39:39.513403 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.513370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:39.517221 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.515235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:39.517221 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.515854 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:39:39.529526 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.527443 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:39:39.537524 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.537497 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:39.757435 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.757411 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wnpsx"] Oct 30 20:39:39.760397 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:39.760369 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc36e1bc_d1aa_4f15_bb34_5971c8c2b74b.slice/crio-a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083 WatchSource:0}: Error finding container a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083: Status 404 returned error can't find the container with id a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083 Oct 30 20:39:39.810942 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.810919 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdndq"] Oct 30 20:39:39.812978 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:39.812951 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb3812a_4ba1_46e5_9433_c9f830ac9629.slice/crio-e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7 WatchSource:0}: Error finding container e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7: Status 404 returned error can't find the container with id e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7 Oct 30 20:39:39.830775 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.830746 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5kr7\"" Oct 30 20:39:39.838855 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:39.838793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:40.053509 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:40.053469 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdndq" event={"ID":"7bb3812a-4ba1-46e5-9433-c9f830ac9629","Type":"ContainerStarted","Data":"e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7"} Oct 30 20:39:40.054958 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:40.054930 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wnpsx" event={"ID":"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b","Type":"ContainerStarted","Data":"a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083"} Oct 30 20:39:40.120619 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:40.120533 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ktb85"] Oct 30 20:39:40.125536 ip-10-0-141-65 kubenswrapper[2580]: W1030 20:39:40.125502 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557f314_88c8_4807_aeb5_6e73cf74a44c.slice/crio-f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700 WatchSource:0}: Error finding container f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700: Status 404 returned error can't find the container with id f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700 Oct 30 20:39:41.061967 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:41.061864 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ktb85" event={"ID":"a557f314-88c8-4807-aeb5-6e73cf74a44c","Type":"ContainerStarted","Data":"f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700"} Oct 30 20:39:42.069912 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:42.069865 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdndq" event={"ID":"7bb3812a-4ba1-46e5-9433-c9f830ac9629","Type":"ContainerStarted","Data":"48f4613eb1caa04f4db80f7bd5d4719785f189769d7841689226492d70d1c432"} Oct 30 20:39:42.070291 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:42.069936 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdndq" event={"ID":"7bb3812a-4ba1-46e5-9433-c9f830ac9629","Type":"ContainerStarted","Data":"080fb328c80d71006f247a3b4f21efb607fc34e8042f4a42e27ba6a85aee8885"} Oct 30 20:39:42.088589 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:42.088377 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bdndq" podStartSLOduration=66.732938135 podStartE2EDuration="1m8.088355163s" totalImagesPullingTime="1.355417028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:42.087528382 +0000 UTC m=+68.953322608" watchObservedRunningTime="2026-10-30 20:39:42.088355163 +0000 UTC m=+68.954149390" Oct 30 20:39:45.085684 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:45.085607 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ktb85" event={"ID":"a557f314-88c8-4807-aeb5-6e73cf74a44c","Type":"ContainerStarted","Data":"8570fef6a86d6d5ef1df01d56f77a90b0341d3139f50a4a22fedfd575c56e363"} Oct 30 20:39:45.086260 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:45.085818 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:45.102864 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:45.102487 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ktb85" podStartSLOduration=66.371578918 podStartE2EDuration="1m11.102467457s" totalImagesPullingTime="4.730888539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:45.10235818 +0000 UTC m=+71.968152406" watchObservedRunningTime="2026-10-30 20:39:45.102467457 +0000 UTC m=+71.968261684" Oct 30 20:39:46.540681 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.540573 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:39:46.544383 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.544352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:46.550814 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.550784 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:39:46.551138 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.551108 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:39:46.551138 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.551128 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:39:46.551315 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.551147 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jspb5\"" Oct 30 20:39:46.551392 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.551312 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:39:46.551446 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.551422 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:39:46.556413 ip-10-0-141-65 kubenswrapper[2580]: I1030 20:39:46.555883 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:39:46.560010 ip-10-0-141-65 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:39:46.567498 ip-10-0-141-65 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:39:46.567748 ip-10-0-141-65 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:39:46.567811 ip-10-0-141-65 systemd[1]: kubelet.service: Consumed 4.186s CPU time. Oct 30 20:39:46.578307 ip-10-0-141-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:39:46.677712 ip-10-0-141-65 kubenswrapper[5855]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:46.677712 ip-10-0-141-65 kubenswrapper[5855]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:39:46.677712 ip-10-0-141-65 kubenswrapper[5855]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:46.677712 ip-10-0-141-65 kubenswrapper[5855]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:46.679952 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679848 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.679952 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679952 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.679952 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679956 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.679952 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679959 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679962 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679965 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679968 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679971 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679973 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679976 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679979 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679981 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679984 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679987 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679990 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679993 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679995 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.679997 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680000 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680007 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680010 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680013 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.680107 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680015 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680018 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680020 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680023 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680026 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680028 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680031 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680033 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680036 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680038 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680041 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680043 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680045 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680048 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680051 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680053 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680056 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680060 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680063 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680065 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.680588 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680068 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680070 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680073 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680075 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680078 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680080 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680083 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680086 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680088 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680091 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680093 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680098 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680102 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680105 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680109 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680111 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680114 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680117 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680120 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.681188 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680123 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680125 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680129 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680132 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680135 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680137 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680140 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680143 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680145 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680148 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680151 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680154 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680156 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680159 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680161 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680164 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680166 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680169 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680172 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680174 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.681716 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680177 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680179 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680182 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680184 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680187 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680191 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680195 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680198 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680202 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680205 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680208 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680211 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680214 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680217 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680219 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680222 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680224 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680227 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680230 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.682232 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680232 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680235 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680237 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680241 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680244 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.680247 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681162 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681171 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681174 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681177 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681185 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681189 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681192 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681196 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681199 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681202 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681205 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681208 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681211 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681213 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.682721 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681216 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681218 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681221 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681223 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681226 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681229 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681232 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681235 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681237 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681240 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681242 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681245 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681248 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681251 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681253 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681257 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681259 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681262 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681266 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681269 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.683228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681272 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681275 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681278 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681281 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681283 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681285 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681288 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681290 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681293 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681296 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681298 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681301 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681303 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681306 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681309 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681312 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681314 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681317 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681319 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681322 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.683759 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681324 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681327 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681329 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681332 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681334 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681337 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681340 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681344 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681347 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681350 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681352 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681356 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681359 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681361 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681364 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681369 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681373 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681376 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681380 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.684261 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681382 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681385 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681387 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681390 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681392 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681395 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681397 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681399 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681402 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681404 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681407 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681410 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681413 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681415 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681417 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681420 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681423 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681426 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681428 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.684774 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681431 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681435 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681439 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681442 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681446 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681449 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681453 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681456 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681459 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681461 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681464 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681466 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681469 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.681471 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682095 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682104 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682108 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682112 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682115 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682118 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.685252 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682121 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682123 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682127 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682129 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682132 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682135 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682137 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682140 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682142 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682145 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682149 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682153 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682156 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682159 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682162 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682164 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682166 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682169 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682172 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.685801 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682175 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682178 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682180 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682183 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682185 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682188 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682190 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682193 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682197 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682200 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682204 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682207 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682209 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682212 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682214 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682220 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682223 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682226 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682229 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682231 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.686275 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682234 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682237 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682239 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682243 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682246 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682248 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682251 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682253 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682256 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682258 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682260 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682263 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682266 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682269 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682272 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682274 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682277 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682279 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682282 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.686902 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682284 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682287 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682289 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682292 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682294 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682297 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682299 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682302 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682307 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682309 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682312 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682314 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682317 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682319 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682322 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682325 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682327 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682329 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682332 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682335 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.687426 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682338 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682340 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682343 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682345 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682348 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682350 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682353 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682355 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682358 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682360 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682363 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682366 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682368 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682371 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682374 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682377 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682379 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682382 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682384 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682387 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.688034 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682393 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.682396 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682507 5855 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682518 5855 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682528 5855 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682535 5855 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682543 5855 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682547 5855 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682551 5855 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682556 5855 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682560 5855 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682563 5855 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682566 5855 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682570 5855 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682573 5855 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682576 5855 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682579 5855 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682582 5855 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682584 5855 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682591 5855 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682596 5855 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682599 5855 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682603 5855 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682606 5855 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:39:46.688869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682609 5855 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682613 5855 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682617 5855 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682620 5855 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682623 5855 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682626 5855 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682629 5855 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682632 5855 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682637 5855 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682640 5855 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682644 5855 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682647 5855 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682664 5855 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682667 5855 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682671 5855 flags.go:64] FLAG: --enable-server="true" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682674 5855 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682677 5855 flags.go:64] FLAG: --event-burst="100" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682681 5855 flags.go:64] FLAG: --event-qps="50" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682683 5855 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682687 5855 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682690 5855 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682694 5855 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682697 5855 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682700 5855 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682703 5855 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:39:46.689697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682706 5855 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682709 5855 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682712 5855 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682717 5855 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682720 5855 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682723 5855 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682726 5855 flags.go:64] FLAG: --feature-gates="" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682731 5855 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682734 5855 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682737 5855 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682741 5855 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682744 5855 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682747 5855 flags.go:64] FLAG: --help="false" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682750 5855 flags.go:64] FLAG: --hostname-override="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682753 5855 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682757 5855 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682760 5855 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682764 5855 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682767 5855 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682770 5855 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682773 5855 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682776 5855 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682779 5855 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:39:46.690334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682781 5855 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682785 5855 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682788 5855 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682790 5855 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682793 5855 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682797 5855 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682799 5855 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682802 5855 flags.go:64] FLAG: --lock-file="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682805 5855 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682807 5855 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682811 5855 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682816 5855 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682821 5855 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682824 5855 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682827 5855 flags.go:64] FLAG: --logging-format="text" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682830 5855 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682833 5855 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682836 5855 flags.go:64] FLAG: --manifest-url="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682839 5855 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682843 5855 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682847 5855 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682850 5855 flags.go:64] FLAG: --max-pods="110" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682853 5855 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682856 5855 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682861 5855 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:39:46.690926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682864 5855 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682868 5855 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682871 5855 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682874 5855 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682882 5855 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682885 5855 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682888 5855 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682891 5855 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682894 5855 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682897 5855 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682900 5855 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682903 5855 flags.go:64] FLAG: --port="10250" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682906 5855 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682909 5855 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0f225ae7ec694b03a" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682912 5855 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682914 5855 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682917 5855 flags.go:64] FLAG: --register-node="true" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682920 5855 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682924 5855 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682928 5855 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682931 5855 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682935 5855 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682939 5855 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682942 5855 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682945 5855 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682948 5855 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:39:46.691534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682951 5855 flags.go:64] FLAG: --runonce="false" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682954 5855 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682957 5855 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682960 5855 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682963 5855 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682967 5855 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682970 5855 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682973 5855 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682976 5855 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682978 5855 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682981 5855 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682984 5855 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682987 5855 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682990 5855 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682993 5855 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.682996 5855 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683001 5855 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683003 5855 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683006 5855 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683010 5855 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683013 5855 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683015 5855 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683018 5855 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683021 5855 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683023 5855 flags.go:64] FLAG: --v="2" Oct 30 20:39:46.692269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683029 5855 flags.go:64] FLAG: --version="false" Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683033 5855 flags.go:64] FLAG: --vmodule="" Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683037 5855 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683041 5855 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683186 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683190 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683194 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683197 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683199 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683202 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683205 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683207 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683211 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683214 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683217 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683219 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683221 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683224 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683226 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683229 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683232 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.692958 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683235 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683237 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683240 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683243 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683245 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683248 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683251 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683253 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683256 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683258 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683260 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683264 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683266 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683269 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683271 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683274 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683277 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683279 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683282 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.693477 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683284 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683286 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683289 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683292 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683295 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683298 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683300 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683302 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683305 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683308 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683311 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683313 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683316 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683318 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683321 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683323 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683326 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683328 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683330 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.694077 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683333 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683335 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683338 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683340 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683343 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683345 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683348 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683350 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683353 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683355 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683359 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683362 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683364 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683367 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683369 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683372 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683374 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683379 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683383 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683385 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.694810 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683387 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683390 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683393 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683395 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683398 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683400 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683403 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683405 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683408 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683410 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683413 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683415 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683417 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683421 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683425 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683427 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683430 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683433 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683437 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683441 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.695450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683443 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683446 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683448 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683451 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683454 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683457 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683459 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683462 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683464 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683468 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683471 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.683496 5855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683532 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683537 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683541 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683544 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.696209 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683547 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683550 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683553 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683555 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683558 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683560 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683563 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683566 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683569 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683572 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683574 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683577 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683580 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683582 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683585 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683588 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683590 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683593 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683596 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683598 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.696735 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683601 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683604 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683607 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683609 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683612 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683614 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683618 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683621 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683623 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683626 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683641 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683663 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683666 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683668 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683671 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683673 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683676 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683678 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683681 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.697248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683684 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683687 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683689 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683691 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683694 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683696 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683699 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683701 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683704 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683706 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683709 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683711 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683713 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683716 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683719 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683722 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683724 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683727 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683729 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683733 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.697742 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683736 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683738 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683741 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683743 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683748 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683750 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683753 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683757 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683760 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683763 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683766 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683768 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683771 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683774 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683776 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683779 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683781 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683784 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683786 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683789 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.698262 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683791 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683794 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683798 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683801 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683804 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683807 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683809 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683814 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683818 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683821 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683823 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683830 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683833 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683836 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683839 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683841 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683845 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683848 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683850 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.698782 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683853 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683856 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683858 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.683861 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.689310 5855 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.689327 5855 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.689346 5855 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.689351 5855 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689421 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689425 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689428 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689431 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689434 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689436 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689439 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689442 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689444 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689447 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689449 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.699267 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689452 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689455 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689457 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689460 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689463 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689466 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689469 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689471 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689474 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689476 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689480 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689482 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689485 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689487 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689490 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689496 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689499 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689502 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689504 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689507 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.699786 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689509 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689512 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689514 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689517 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689519 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689522 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689524 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689527 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689530 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689532 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689535 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689538 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689540 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689543 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689546 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689550 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689552 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689555 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689557 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689560 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.700298 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689563 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689565 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689567 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689570 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689573 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689575 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689578 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689580 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689583 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689585 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689588 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689590 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689593 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689595 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689598 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689600 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689603 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689605 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689608 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.701122 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689610 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689613 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689617 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689622 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689625 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689627 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689630 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689632 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689635 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689638 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689640 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689643 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689645 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689648 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689678 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689682 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689685 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689688 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.701622 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689690 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689693 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689695 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689699 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689701 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689704 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689706 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689709 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689711 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689714 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689717 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689719 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689722 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689724 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689726 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689729 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689731 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.702106 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689733 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.689766 5855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689922 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689927 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689929 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689932 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689935 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689938 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689941 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689944 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689946 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689949 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689951 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689953 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689956 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689958 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.702601 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689961 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689963 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689966 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689969 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689971 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689974 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689977 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689979 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689982 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689984 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689987 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689990 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689992 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689994 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689997 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.689999 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690002 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690005 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690007 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690010 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.703165 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690012 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690015 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690018 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690020 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690023 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690025 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690028 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690030 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690033 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690035 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690038 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690041 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690043 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690045 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690048 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690051 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690053 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690056 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690058 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.703828 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690061 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690063 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690066 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690068 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690071 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690073 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690076 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690078 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690081 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690083 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690086 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690089 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690093 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690096 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690099 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690101 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690103 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690106 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690109 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.704463 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690112 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690114 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690117 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690119 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690122 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690124 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690127 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690129 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690131 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690134 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690136 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690139 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690142 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690145 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690148 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690151 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690154 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690157 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690159 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.705112 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690162 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690164 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690167 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690169 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690171 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690174 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690176 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690179 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690181 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690183 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690186 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690191 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690193 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690196 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690199 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.690225 5855 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:46.705606 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690257 5855 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690262 5855 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690265 5855 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690268 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690273 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690276 5855 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690279 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690281 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690284 5855 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690286 5855 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690289 5855 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690292 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690294 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690297 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690299 5855 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690302 5855 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690305 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690307 5855 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690310 5855 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690312 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:46.706147 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690314 5855 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690317 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690320 5855 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690323 5855 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690325 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690328 5855 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690330 5855 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690333 5855 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690336 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690339 5855 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690347 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690350 5855 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690352 5855 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690355 5855 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690357 5855 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690360 5855 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690364 5855 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690366 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690369 5855 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:46.706648 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690371 5855 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690374 5855 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690376 5855 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690379 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690382 5855 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690384 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690387 5855 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690389 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690392 5855 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690395 5855 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690397 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690401 5855 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690405 5855 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690408 5855 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690410 5855 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690413 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690415 5855 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690417 5855 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690420 5855 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:46.707153 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690422 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690425 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690429 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690432 5855 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690434 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690438 5855 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690440 5855 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690443 5855 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690446 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690448 5855 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690452 5855 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690454 5855 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690458 5855 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690461 5855 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690464 5855 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690467 5855 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690470 5855 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690472 5855 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690475 5855 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:46.707730 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690478 5855 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690480 5855 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690482 5855 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690485 5855 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690487 5855 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690490 5855 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690492 5855 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690495 5855 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690497 5855 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690500 5855 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690502 5855 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690505 5855 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690507 5855 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690509 5855 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690512 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690516 5855 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690518 5855 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690521 5855 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690523 5855 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690526 5855 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690529 5855 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:46.708303 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690531 5855 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690534 5855 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690538 5855 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690540 5855 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690543 5855 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690545 5855 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690548 5855 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:46.690551 5855 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.690620 5855 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.691817 5855 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.691890 5855 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.692101 5855 server.go:1014] "Starting client certificate rotation" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.692247 5855 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.692308 5855 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:33 +0000 UTC" deadline="2028-07-01 11:17:05.594043559 +0000 UTC" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.692338 5855 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14630h37m18.901707772s" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.692632 5855 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:39:46.709205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.693867 5855 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:39:46.709883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.695505 5855 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:39:46.709883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.698005 5855 log.go:25] "Validated CRI v1 image API" Oct 30 20:39:46.709883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.698942 5855 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:39:46.709883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.701505 5855 fs.go:135] Filesystem UUIDs: map[550086cb-f288-4101-bc81-e1c3af8f96f7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fce9c592-0ba1-40c5-a991-a65b31c4e745:/dev/nvme0n1p4] Oct 30 20:39:46.710015 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.701538 5855 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/539f653922078e65310cb77c26fc1fc81706df4575fb4ab1d8a6332a51c0743d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/539f653922078e65310cb77c26fc1fc81706df4575fb4ab1d8a6332a51c0743d/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f7f4eb58565f909e35ec5593293e41a8f48e100fb49663c20e51849bc23e0e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f7f4eb58565f909e35ec5593293e41a8f48e100fb49663c20e51849bc23e0e5/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8308c9586783f8ef39ec3b10e5dbf805a5ceb1cd3dd9ab5f209a181446e36c93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8308c9586783f8ef39ec3b10e5dbf805a5ceb1cd3dd9ab5f209a181446e36c93/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94028320deb5796dddac04dd4dcc42ffcdb698130df2af2e21f796ceff03383d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94028320deb5796dddac04dd4dcc42ffcdb698130df2af2e21f796ceff03383d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01f5a5d4-d276-45f8-869e-a41dbd0d445b/volumes/kubernetes.io~projected/kube-api-access-s5fw5:{mountpoint:/var/lib/kubelet/pods/01f5a5d4-d276-45f8-869e-a41dbd0d445b/volumes/kubernetes.io~projected/kube-api-access-s5fw5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~projected/kube-api-access-km96v:{mountpoint:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~projected/kube-api-access-km96v major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b22b136-e779-4470-b5ab-b894512f0294/volumes/kubernetes.io~projected/kube-api-access-dbfq4:{mountpoint:/var/lib/kubelet/pods/0b22b136-e779-4470-b5ab-b894512f0294/volumes/kubernetes.io~projected/kube-api-access-dbfq4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1565be2d-9663-42b1-bc86-72e4263fe7c3/volumes/kubernetes.io~projected/kube-api-access-fwx7w:{mountpoint:/var/lib/kubelet/pods/1565be2d-9663-42b1-bc86-72e4263fe7c3/volumes/kubernetes.io~projected/kube-api-access-fwx7w major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~projected/kube-api-access-c4thr:{mountpoint:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~projected/kube-api-access-c4thr major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~projected/kube-api-access-tfqpw:{mountpoint:/var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~projected/kube-api-access-tfqpw major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~projected/kube-api-access-xwv46:{mountpoint:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~projected/kube-api-access-xwv46 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3039da16-1889-4c78-b2c0-4b612a0cb125/volumes/kubernetes.io~projected/kube-api-access-rmnht:{mountpoint:/var/lib/kubelet/pods/3039da16-1889-4c78-b2c0-4b612a0cb125/volumes/kubernetes.io~projected/kube-api-access-rmnht major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~projected/kube-api-access-4rcqv:{mountpoint:/var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~projected/kube-api-access-4rcqv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~projected/kube-api-access-hkbt9:{mountpoint:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~projected/kube-api-access-hkbt9 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~projected/kube-api-access-rzx65:{mountpoint:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~projected/kube-api-access-rzx65 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~projected/kube-api-access-9rng4:{mountpoint:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~projected/kube-api-access-9rng4 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51cdc667-f01e-435d-8cf8-d7e805e0efd6/volumes/kubernetes.io~projected/kube-api-access-55kjl:{mountpoint:/var/lib/kubelet/pods/51cdc667-f01e-435d-8cf8-d7e805e0efd6/volumes/kubernetes.io~projected/kube-api-access-55kjl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54acb4d1-67b7-4ac8-9884-11af4f940626/volumes/kubernetes.io~projected/kube-api-access-4j9m8:{mountpoint:/var/lib/kubelet/pods/54acb4d1-67b7-4ac8-9884-11af4f940626/volumes/kubernetes.io~projected/kube-api-access-4j9m8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~projected/kube-api-access-dnnm4:{mountpoint:/var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~projected/kube-api-access-dnnm4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~secret/metrics-certs major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~projected/kube-api-access-mf98h:{mountpoint:/var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~projected/kube-api-access-mf98h major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bd09df0-d50b-4f00-863d-276efe60e695/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8bd09df0-d50b-4f00-863d-276efe60e695/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~projected/kube-api-access-92vcm:{mountpoint:/var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~projected/kube-api-access-92vcm major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93073bfa-1168-4c11-a950-1f75458694d1/volumes/kubernetes.io~projected/kube-api-access-csdjh:{mountpoint:/var/lib/kubelet/pods/93073bfa-1168-4c11-a950-1f75458694d1/volumes/kubernetes.io~projected/kube-api-access-csdjh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a557f314-88c8-4807-aeb5-6e73cf74a44c/volumes/kubernetes.io~projected/kube-api-access-s8vnf:{mountpoint:/var/lib/kubelet/pods/a557f314-88c8-4807-aeb5-6e73cf74a44c/volumes/kubernetes.io~projected/kube-api-access-s8vnf major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~projected/kube-api-access-2gl9h:{mountpoint:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~projected/kube-api-access-2gl9h major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:358 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6b8b6df144119b5395710e583835d462f575400dfc2b24b0fd82c5d40ffe5b32/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c68840b3128bb17324f7e676ac4e0e11c860a88893b7ff569c8b4721c78fcb11/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3a31572bb60fb3db989eaa1145e0c5b7b313dcbb54cfc1241bba86cc11e219fc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/dfcf3425ad6786907a1a298b73fe179eee2801a2b62242ce0b4a91cf5957aa5f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9dbcdb265e9e3f24d031db5ff9241399dbb588fc812b95bfe7e55759e17cb90a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/01e8db3d3a737372761ab44b1028efb67ca2343d6944d84486999aa93f070c1a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/247e801fb23616f7e20f6f12b08f16e05c2360e6dda17f1900f30bd3f5b21a7b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6e15583036121d03c7d184ace0191f4701a59754f935633d8d31edf2862ac733/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cce3d7a1d6ad95bfa90619419d5a5b156ab231bca6f29455299105713a5d3679/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b548417d0022d748b3d40fc5cae86a98082f2379de73df906104d9b0f2e69d12/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/94bf8cb94b75ef836194db2ff230acf6b76451c8fe87267599e1d9f981c3089d/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/17a3ffcd86aced02cbd97220da27c52b7a5c6b83752d61e5d74b314c8a726b34/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/34d203a4009a017a01cbfef3581707cd0f5bfce2475c6bafb89e43d0b53bb01e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/074107f9426ad327c8511ecaaab77d9a86cd80a5368b70dfb0813413c4a1f92e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/a818598e55e7bc314dfc5bd0e6f80b670c245bfaa525fc402f1d713768c464ff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9b3ac4ba0ed01349588e63f66d2716f4dd7c6e018646d8cf68dc6e6dc1e3cb99/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/acae58cd542c46e3e3956ba514a2ac2e2d3dd15b507fb1cc869d7ac0f76da902/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9ab9d835ed049523eecae1ab31560a4d8859c29f7150679e204307bde35eb553/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4b936fcb88550fa5d0337aca0700c782b1f1d261a4f3d4c55e4dd3688436de34/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0648d242856ba7d6a18a34a3a1277545da62415b41581ac58600f69ba48c7df6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1c8e7cae714c5463ba9f53a72b22813896adc754daa12ecf79a1dc70db182852/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/e4fb16e02517a76b83e4e484bf80d10787521cfc2443f5a157da223ac7aa7eed/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/059d234c197b729aaab6ae3ec55116d244d8f94e59c291e540405d54bf6d3a3d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9d3f68c63183ffbd6bd3f759d0472c34b85dc0a82d61b99b138f39c5dc8ae576/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5d3b108677fe09e6b39ea4de602d89b9f8500b68d106961de1f1724d2648bc7a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/25b1b41255284c5ed1315f43649dc7e75a7b2e35b334c8d6f4c2ea02af508fca/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/a844a8fe9a4326d28337a14d89d15d7abab0c56fdb5b090bdda418d767266dc3/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/c7d02eab231344dbf136cf63ccd94f20bcffa213a407602a63c5dd0e005c1f66/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/d06a2d1464050fe1a43b1e6008ebb61effcf667e629013e11de5ff5bbf5b8210/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/d00980102aab8bfe2926a955c7adee8919ef2b71ac874b58b3d098130ea53a3a/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/8693e5d33595ba599867b26d0b75daa462efe2e97ca63e03471b3842b47d3506/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b5620b325f17761136295c58af1b82211822d8aa28174731df3ea89cc33c6880/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/621a5d4faaca1bc20399631cc46b6bed304835d5d04f64e3e0b261bfee58fc61/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/0b23571eb269051c01ea34726e180c2e4cba0628e304eded31a5c679930eb325/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/612a06a185faf700736508e46b07a6c067783ed705fe3e4db2bff7a1edad184e/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/c08decb25856fe4ab7ac8e0f6e9237d17c858a7f5d632b6ec9feb8e75c5d5cfc/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/1d1b706aa1c794048d6d39a66f67ac54abc007d692c530e1bc561ad57e42ecc8/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/b143b75c6695b9cbadaf5cfd324b7cd2736859db2327617571359018ddea69cf/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/f2b1e84e5b4e395abcc6c4fbf664c6077b9914ae0089852f43e0a6507105438c/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/71920abcc487a9b73141d487dba9eba9231b6c2126d1469c8bfaf0b19d16600d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/d7a936c7e5d9ff50a90e5be8ef303bd305417a36f0f637c2e2f39e67a5615bdd/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/a54ce3800e91027817d9f6a5198881bcd065bc10fa076bcb910f828f4d77a8ce/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/2a9f57b854291e0c697ccf2dc024016ceb1981ad10cd9feae653bb8e811e420a/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/5c93a055a51aaa1b4a4bf124143aa79dae00f7315b047d847a1d45e473b2f25a/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/4caf388330a42bd665a5c31af9138e153dcbb5b49a1c6f0288a518a467084489/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/6b5e9350ddb8f0a7512c0474454033c1071a80723790e3ec8dd27677464532be/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/4e285f64c0fdb38ee8551b90f64cd76fdd0803f509e2e0127ebae37c161a9bcc/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/d637b8ca4c49b2716d47e0e76630b45e1e24b985899e0effd9001084d31e43e4/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/867272e7c9370836aaa0d4d1222476c7fbde6fa2598cc055dc1dd686c119ef9c/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/f7b01c2aa53424279d227b2ea2ed22dfd537893e593f26d772e6f1e966692f94/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/e22c0306ce67d4422f1e96d4a402f015d050f3931198f1db2a50a964bb5a5cc8/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/6736a52c1a4211bbbd278c58941fdacb36bda45d5d05f5c4f3b3f9d13583418d/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/792e32b4d855a69c8eee02c2d306abbe7c9f992370b16eb91baa2aa13e50221c/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/aee7ee62064c61ba5520a7aba880520334d406d57e366096d9ecd797b1cbd213/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/04c9bcd6694619c95eaa0de8f59ed44534207b93f85b8b0da105ff30c60cfba4/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/e8f1b2db6432d4620e6bb4f9e46d17ff876fb1c7f1f1c1bb8b6af6b34afd84ca/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/beb9a402c07fb04ace0a28c590114cbb0a2e349de656d08dbf0265027ed2c4c0/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/58b037cc0e7077a557ed010630e2e687a0ca0b8c4a9e99b71e008fc53026aca0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d4cc4763a18c552570612895f42fe044e2e757a5ae7c5e0c7f91589904487c2e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f1010037e005dd48d4027ea3facdb328d3020ce90bffe8591b5764f64615b5b2/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/187b2103a94ec27953c9991cb499d973d0cc3321bd7b77266ed7777dbb844ff3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/18faccf7cd0bb185a8bd758c7e119095d29ad04a5ed596833b3a5633ffc8404f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7081834965b9bebaf4a5d2c76c151a876f1a99446c891e2069737ddd4720f08e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c5a08398dfda735a1f6e9ecb9387e570d8c9f3735a3a40c869c0824e5bde4bee/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/fd4fae13ac8346558051ceb46f059441922e6886e6d01e6826e990504034ab87/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/61c2df4728aa773b6eb5e553e77254b5b39880b7c07b7b6847f82c9caf8ee8be/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 20:39:46.711319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.710918 5855 manager.go:217] Machine: {Timestamp:2026-10-30 20:39:46.710336227 +0000 UTC m=+0.069048964 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094671 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bcc768e77836fda87ad7dc98f998 SystemUUID:ec24bcc7-68e7-7836-fda8-7ad7dc98f998 BootID:6a7b7e2c-fd7f-4a19-9f27-118b7aced702 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9360ab1a27b46008c3c5ceb410eb19429f6c859360e83b15eb9874bbbe400a7/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/51cdc667-f01e-435d-8cf8-d7e805e0efd6/volumes/kubernetes.io~projected/kube-api-access-55kjl DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~projected/kube-api-access-tfqpw DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d08f587258212b099c6428cae429f7825bb27d25859bee6711ba5226dfe42556/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1640dd96be2deaead0ab89a00985ee4c78a9284ce7fab8fb0f8e85d21b3e65a4/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~projected/kube-api-access-c4thr DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/845a61a529e78349d8dfc41464aca93ae7e05734bd38634109e26caf7bbd4a57/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e93775faa3e71b3df3d6d86496506949d3d35f46ec8d703a08cca4da9beb9d75/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1565be2d-9663-42b1-bc86-72e4263fe7c3/volumes/kubernetes.io~projected/kube-api-access-fwx7w DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b22b136-e779-4470-b5ab-b894512f0294/volumes/kubernetes.io~projected/kube-api-access-dbfq4 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22c7dfffd561674e0a038c706e39bbaae78f07601fa97766d074bd32b583b3d7/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~projected/kube-api-access-2gl9h DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~projected/kube-api-access-mf98h DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~projected/kube-api-access-xwv46 DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~projected/kube-api-access-dnnm4 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~projected/kube-api-access-92vcm DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~projected/kube-api-access-km96v DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8005a7fb10fcad1af866cbdd0415df54c560c2a9b5f2297bdee640c2bae5700/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bd09df0-d50b-4f00-863d-276efe60e695/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29b8796df33b64dcdc2c505585daa6f57a2e5d2a91607522eac148eb72c4c757/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~projected/kube-api-access-rzx65 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16f2b5e0-aaed-456d-a023-c2bbf64765dd/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47a8078d-c222-4943-af36-4285c0b450cd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:261 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18931dbdc262b9b478e3a78a0a88efed9b1b9e26b2e3e26132cceedd91789006/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9f8a077040db0f32bff76a952179445357182a9932d6aa8fe94076005676083/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/01f5a5d4-d276-45f8-869e-a41dbd0d445b/volumes/kubernetes.io~projected/kube-api-access-s5fw5 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/539f653922078e65310cb77c26fc1fc81706df4575fb4ab1d8a6332a51c0743d/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a21bfa7bcf073b62ac0a0a463aa15b7e7b12e4bb8480a4235bffa10cdfa9e330/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8308c9586783f8ef39ec3b10e5dbf805a5ceb1cd3dd9ab5f209a181446e36c93/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54acb4d1-67b7-4ac8-9884-11af4f940626/volumes/kubernetes.io~projected/kube-api-access-4j9m8 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84b5c8b5a09e4d63d1d7b1b1e335be84d719f79a77322cd72f042d8fceb1f1db/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0208ffae-7118-4175-8e46-97ac0592ebfc/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:358 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a557f314-88c8-4807-aeb5-6e73cf74a44c/volumes/kubernetes.io~projected/kube-api-access-s8vnf DeviceMajor:0 DeviceMinor:368 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e1a4f94ee3471b7b064b64ce3244eb413996de5d0bcad662b8ae1767472c058/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a338f845f651c0246c8a0c29915676d83969020000bb46a64307dbe5583c8e17/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6dcce165170ac0cdb9cc0cfa4b075ed91b728d590e349ba48be96e4275b6734/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8866b225-b24b-43f3-b2a5-960fa1d9f751/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1912739c-4112-436d-9000-e5858dce9aeb/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dcbbaec00c79f42dff5481008e92717731b9e94f3a88ba1201e3ec9119eaf24/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3960ed57-c28a-413f-817e-355fa6029130/volumes/kubernetes.io~projected/kube-api-access-4rcqv DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4eca690e-c8c5-4526-bdc4-7cb0a436fc30/volumes/kubernetes.io~projected/kube-api-access-9rng4 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3039da16-1889-4c78-b2c0-4b612a0cb125/volumes/kubernetes.io~projected/kube-api-access-rmnht DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7bb3812a-4ba1-46e5-9433-c9f830ac9629/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:363 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f744cc86-1e11-4932-894f-8610acdcff4e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93073bfa-1168-4c11-a950-1f75458694d1/volumes/kubernetes.io~projected/kube-api-access-csdjh DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/26071f56-69e8-4327-a05f-deaf1f8e209a/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:271 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac2e542-da70-4445-a126-d02866a7e2f9/volumes/kubernetes.io~projected/kube-api-access-hkbt9 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94028320deb5796dddac04dd4dcc42ffcdb698130df2af2e21f796ceff03383d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f7f4eb58565f909e35ec5593293e41a8f48e100fb49663c20e51849bc23e0e5/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8706e086399f3e97c610d608c1279be6a642d07482848766b9224a5e638a1383/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1640dd96be2deae MacAddress:32:5e:53:f2:f5:c1 Speed:10000 Mtu:8901} {Name:18931dbdc262b9b MacAddress:ee:c2:e3:fd:91:c3 Speed:10000 Mtu:8901} {Name:22c7dfffd561674 MacAddress:f2:76:3c:38:e1:80 Speed:10000 Mtu:8901} {Name:4e1a4f94ee3471b MacAddress:3a:0c:b4:00:bf:42 Speed:10000 Mtu:8901} {Name:84b5c8b5a09e4d6 MacAddress:c6:31:80:37:4b:57 Speed:10000 Mtu:8901} {Name:8706e086399f3e9 MacAddress:d6:73:ea:f1:5c:02 Speed:10000 Mtu:8901} {Name:a21bfa7bcf073b6 MacAddress:c2:1d:8c:38:94:76 Speed:10000 Mtu:8901} {Name:a6dcce165170ac0 MacAddress:62:26:67:04:eb:83 Speed:10000 Mtu:8901} {Name:a9f8a077040db0f MacAddress:76:1b:65:eb:9f:ed Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ef:ab:86:51 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:35:7a:e6:0d:50 Speed:0 Mtu:8901} {Name:e9360ab1a27b460 MacAddress:6a:4a:bd:44:e1:20 Speed:10000 Mtu:8901} {Name:e93775faa3e71b3 MacAddress:5a:44:02:d6:b2:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ef:ab:86:51 Speed:0 Mtu:9001} {Name:f8005a7fb10fcad MacAddress:16:f1:b6:49:76:22 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:0d:ec:aa:54:fc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:d9:c3:ac:86:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:39:46.711504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711323 5855 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:39:46.711504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711378 5855 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:39:46.711957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711746 5855 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:39:46.711957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711776 5855 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-65.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:39:46.711957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711923 5855 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:39:46.711957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711933 5855 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:39:46.711957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711949 5855 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:39:46.712196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.711964 5855 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:39:46.712196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712100 5855 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:39:46.712314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712300 5855 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:39:46.712438 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712427 5855 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:39:46.712492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712482 5855 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:39:46.712524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712501 5855 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:39:46.712524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712513 5855 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:39:46.712524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712523 5855 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:39:46.712612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.712538 5855 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:39:46.713528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.713510 5855 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:39:46.713528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.713526 5855 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:39:46.714247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.714229 5855 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:39:46.724442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.724412 5855 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:39:46.724442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.724435 5855 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:39:46.724721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.724578 5855 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:39:46.724902 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.724883 5855 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:39:46.724964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.724926 5855 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:39:46.725101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725088 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:39:46.725165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725119 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:39:46.725165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725136 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:39:46.725165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725148 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:39:46.725165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725164 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725182 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725192 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725202 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725215 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725224 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725247 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725261 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725302 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:39:46.725338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.725312 5855 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:39:46.729233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.729206 5855 server.go:1263] "Started kubelet" Oct 30 20:39:46.729377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.729304 5855 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:39:46.730118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.729968 5855 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:39:46.730529 ip-10-0-141-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:39:46.731341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.730089 5855 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:39:46.731909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.731839 5855 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:39:46.735000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.733906 5855 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:39:46.735996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.735975 5855 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736243 5855 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736260 5855 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736288 5855 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:46.736334 5855 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-65.us-east-2.compute.internal\" not found" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736457 5855 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736519 5855 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:35 +0000 UTC" deadline="2028-05-20 06:29:15.418018701 +0000 UTC" Oct 30 20:39:46.736999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.736563 5855 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13617h49m28.681459411s" Oct 30 20:39:46.739401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739329 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47a8078d-c222-4943-af36-4285c0b450cd" volumeName="kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned" seLinuxMountContext="" Oct 30 20:39:46.739401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739395 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8866b225-b24b-43f3-b2a5-960fa1d9f751" volumeName="kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert" seLinuxMountContext="" Oct 30 20:39:46.739401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739411 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1565be2d-9663-42b1-bc86-72e4263fe7c3" volumeName="kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739435 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739447 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb3812a-4ba1-46e5-9433-c9f830ac9629" volumeName="kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739461 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01f5a5d4-d276-45f8-869e-a41dbd0d445b" volumeName="kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739483 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3960ed57-c28a-413f-817e-355fa6029130" volumeName="kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739495 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739509 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eca690e-c8c5-4526-bdc4-7cb0a436fc30" volumeName="kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739531 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22b136-e779-4470-b5ab-b894512f0294" volumeName="kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739545 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47a8078d-c222-4943-af36-4285c0b450cd" volumeName="kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739560 5855 factory.go:55] Registering systemd factory Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739568 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47a8078d-c222-4943-af36-4285c0b450cd" volumeName="kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739579 5855 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739591 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4" volumeName="kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739603 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f744cc86-1e11-4932-894f-8610acdcff4e" volumeName="kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:39:46.739758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739618 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01f5a5d4-d276-45f8-869e-a41dbd0d445b" volumeName="kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739814 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0208ffae-7118-4175-8e46-97ac0592ebfc" volumeName="kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739834 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739850 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739864 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3039da16-1889-4c78-b2c0-4b612a0cb125" volumeName="kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739875 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a557f314-88c8-4807-aeb5-6e73cf74a44c" volumeName="kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739888 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1912739c-4112-436d-9000-e5858dce9aeb" volumeName="kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739900 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3960ed57-c28a-413f-817e-355fa6029130" volumeName="kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739912 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3960ed57-c28a-413f-817e-355fa6029130" volumeName="kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739925 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739938 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54acb4d1-67b7-4ac8-9884-11af4f940626" volumeName="kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739949 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93073bfa-1168-4c11-a950-1f75458694d1" volumeName="kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739963 5855 factory.go:153] Registering CRI-O factory Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739979 5855 factory.go:223] Registration of the crio container factory successfully Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.739963 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740049 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740066 5855 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740068 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eca690e-c8c5-4526-bdc4-7cb0a436fc30" volumeName="kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740080 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bd09df0-d50b-4f00-863d-276efe60e695" volumeName="kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740101 5855 factory.go:103] Registering Raw factory Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740121 5855 manager.go:1199] Started watching for new ooms in manager Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740133 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3960ed57-c28a-413f-817e-355fa6029130" volumeName="kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740147 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bd09df0-d50b-4f00-863d-276efe60e695" volumeName="kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740199 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4" volumeName="kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740220 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93073bfa-1168-4c11-a950-1f75458694d1" volumeName="kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740242 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01f5a5d4-d276-45f8-869e-a41dbd0d445b" volumeName="kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740253 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740278 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b" volumeName="kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740290 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740301 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eca690e-c8c5-4526-bdc4-7cb0a436fc30" volumeName="kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740324 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0208ffae-7118-4175-8e46-97ac0592ebfc" volumeName="kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740336 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0208ffae-7118-4175-8e46-97ac0592ebfc" volumeName="kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740347 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46" seLinuxMountContext="" Oct 30 20:39:46.740370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740382 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51cdc667-f01e-435d-8cf8-d7e805e0efd6" volumeName="kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740405 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54acb4d1-67b7-4ac8-9884-11af4f940626" volumeName="kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740416 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4" volumeName="kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740428 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f744cc86-1e11-4932-894f-8610acdcff4e" volumeName="kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740440 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f744cc86-1e11-4932-894f-8610acdcff4e" volumeName="kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740451 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740462 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740474 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740485 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740498 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bb3812a-4ba1-46e5-9433-c9f830ac9629" volumeName="kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740509 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f744cc86-1e11-4932-894f-8610acdcff4e" volumeName="kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740521 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1565be2d-9663-42b1-bc86-72e4263fe7c3" volumeName="kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740532 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0208ffae-7118-4175-8e46-97ac0592ebfc" volumeName="kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740545 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740568 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f2b5e0-aaed-456d-a023-c2bbf64765dd" volumeName="kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740580 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1912739c-4112-436d-9000-e5858dce9aeb" volumeName="kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740611 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3039da16-1889-4c78-b2c0-4b612a0cb125" volumeName="kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740624 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93073bfa-1168-4c11-a950-1f75458694d1" volumeName="kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740638 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93073bfa-1168-4c11-a950-1f75458694d1" volumeName="kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740693 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740708 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eca690e-c8c5-4526-bdc4-7cb0a436fc30" volumeName="kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740723 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8866b225-b24b-43f3-b2a5-960fa1d9f751" volumeName="kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740750 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4" volumeName="kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740765 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f744cc86-1e11-4932-894f-8610acdcff4e" volumeName="kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740781 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22b136-e779-4470-b5ab-b894512f0294" volumeName="kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740794 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26071f56-69e8-4327-a05f-deaf1f8e209a" volumeName="kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740807 5855 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac2e542-da70-4445-a126-d02866a7e2f9" volumeName="kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740820 5855 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740829 5855 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:39:46.741532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.740511 5855 manager.go:319] Starting recovery of all containers Oct 30 20:39:46.744069 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.744042 5855 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:46.744343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.744323 5855 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:46.751236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.749737 5855 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:46.751236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.751025 5855 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:39:46.759832 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.759770 5855 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:39:46.771114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.771068 5855 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:39:46.771114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.771113 5855 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:39:46.771336 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.771234 5855 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:39:46.771336 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:46.771285 5855 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:39:46.777750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.777717 5855 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786353 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="42b146a08d4d1cf4bea73cfb408f31b66f3c7d9b9a8b2147590c47b93db41851" exitCode=0 Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786382 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="29da43e242d395f60a93e802bf4fd9c212197b0bbb480e28e3b62bc71aa200a0" exitCode=0 Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786388 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="74e7140056faf87d4b876d5651eb807292bf74bc71d4c641186f6c15ad08d62b" exitCode=0 Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786394 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="877966eacb4b5201adae3281b40e8093d5c0748ffd5d6449aa40d20647fb8472" exitCode=0 Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786399 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="626e9370e1fac73f99f91bcbcf94222d5d8fa8835ba339c3c839383d7d368854" exitCode=0 Oct 30 20:39:46.786390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.786405 5855 generic.go:356] "Generic (PLEG): container finished" podID="93073bfa-1168-4c11-a950-1f75458694d1" containerID="03a1cb271ea24c87d172c50a0999533360e918fe4cc0fddc2056338cd7b49ef7" exitCode=0 Oct 30 20:39:46.820437 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.820399 5855 generic.go:356] "Generic (PLEG): container finished" podID="16f2b5e0-aaed-456d-a023-c2bbf64765dd" containerID="d836cd9fef8d0481ab02d867a3a8f15d6a986b99b4682cd188ea949fb6104ac5" exitCode=0 Oct 30 20:39:46.829569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.829528 5855 generic.go:356] "Generic (PLEG): container finished" podID="999d3d59afc4db3f7e0da998760ec141" containerID="774789a8b22300550f51b3cad7afe8e11414d7e45c445212e918ec786355797b" exitCode=0 Oct 30 20:39:46.843952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.843931 5855 manager.go:324] Recovery completed Oct 30 20:39:46.866394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866353 5855 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:39:46.866394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866386 5855 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:39:46.866599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866412 5855 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:39:46.866599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866589 5855 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866601 5855 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866621 5855 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866627 5855 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866632 5855 policy_none.go:50] "Start" Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866646 5855 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:39:46.866748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.866670 5855 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:39:46.867521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.867494 5855 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:39:46.867521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.867512 5855 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:39:46.867521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.867520 5855 policy_none.go:44] "Start" Oct 30 20:39:46.871398 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:46.871377 5855 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 20:39:46.873113 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.872516 5855 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:39:46.878170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878157 5855 manager.go:341] "Starting Device Plugin manager" Oct 30 20:39:46.878245 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:46.878183 5855 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:39:46.878245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878192 5855 server.go:87] "Starting device plugin registration server" Oct 30 20:39:46.878592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878578 5855 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:39:46.878644 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878594 5855 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:39:46.878740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878718 5855 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:39:46.878863 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878831 5855 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:39:46.878929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878880 5855 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:39:46.878929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878888 5855 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:39:46.879011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.878938 5855 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:39:46.878861385Z","UUID":"8e43b975-fb8a-4afe-b957-d33b52f33a02","Handler":null,"Name":"","Endpoint":""} Oct 30 20:39:46.880454 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.880437 5855 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:39:46.880552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.880475 5855 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:39:46.880552 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:46.880481 5855 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:39:46.979557 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.979522 5855 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:39:46.981695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.981674 5855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:39:46.981846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.981706 5855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:39:46.981846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.981718 5855 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-65.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:39:46.981846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.981760 5855 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:46.995415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.995392 5855 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:46.995483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:46.995476 5855 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.072180 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072055 5855 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal"] Oct 30 20:39:47.072760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072725 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"f7ada5c68d422440bbf7fd2285220de0","Type":"ContainerStarted","Data":"bde3b6280409d8b27340dece405c918aa545c2f4571664b8a92adc67647b16eb"} Oct 30 20:39:47.072825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072772 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"f7ada5c68d422440bbf7fd2285220de0","Type":"ContainerStarted","Data":"e7c153387d522b7fb7ff2e037f10d87c1d1bd97d966fc76f9f5f9b440cb7daba"} Oct 30 20:39:47.072825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072812 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerStarted","Data":"8aed3c7ce6ebdd998009d0f1c81478a430b77647832416b79425dcf823548948"} Oct 30 20:39:47.072825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072816 5855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.072946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072872 5855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.072946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072825 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerDied","Data":"774789a8b22300550f51b3cad7afe8e11414d7e45c445212e918ec786355797b"} Oct 30 20:39:47.072946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.072922 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" event={"ID":"999d3d59afc4db3f7e0da998760ec141","Type":"ContainerStarted","Data":"6e1c8a3509cea35419d5da486cc2fa1b73c93b6f4c05d33a5d6be71a665424b1"} Oct 30 20:39:47.092111 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.092088 5855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:47.092178 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:47.092155 5855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.092527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.092513 5855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:47.092562 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:47.092551 5855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.152971 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.152934 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.152971 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.152971 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.153150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.152987 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253614 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253577 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253614 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253525 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7ada5c68d422440bbf7fd2285220de0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"f7ada5c68d422440bbf7fd2285220de0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253816 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253639 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253816 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253679 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253816 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253708 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.253816 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.253710 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/999d3d59afc4db3f7e0da998760ec141-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" (UID: \"999d3d59afc4db3f7e0da998760ec141\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.713294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.713258 5855 apiserver.go:52] "Watching apiserver" Oct 30 20:39:47.723148 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.723115 5855 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:39:47.723743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.723719 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz","openshift-ovn-kubernetes/ovnkube-node-z55gr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz","openshift-cluster-node-tuning-operator/tuned-vbqx9","openshift-insights/insights-runtime-extractor-wm9wk","openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb","openshift-console/downloads-6dd789c8b7-gzv86","openshift-image-registry/node-ca-5cmmq","openshift-multus/multus-additional-cni-plugins-gd9bl","kube-system/konnectivity-agent-ll9wf","kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal","openshift-console/console-7f69955664-z6kgn","openshift-dns/dns-default-m74jp","openshift-multus/network-metrics-daemon-bdndq","openshift-dns/node-resolver-bfbs4","openshift-monitoring/metrics-server-656d8b8c49-pmmgw","openshift-monitoring/monitoring-plugin-77f4f76594-dck4k","kube-system/global-pull-secret-syncer-wnpsx","openshift-monitoring/node-exporter-xx98g","openshift-network-diagnostics/network-check-target-ktb85","openshift-network-operator/iptables-alerter-x874n","openshift-ingress-canary/ingress-canary-jggzv","openshift-multus/multus-65qrq","openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal"] Oct 30 20:39:47.729173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.729154 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.729601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.729571 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.729601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.729597 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.730090 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.730071 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:39:47.734671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.734633 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.734991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.734971 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.735071 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735047 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:39:47.735124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735101 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.735185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735174 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hpn29\"" Oct 30 20:39:47.735327 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735303 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.735433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735323 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:39:47.735433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735307 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.735433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735343 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4khcx\"" Oct 30 20:39:47.735574 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.735516 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.736204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736186 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:39:47.736293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736246 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.736293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736271 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:39:47.736387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736342 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hh9qp\"" Oct 30 20:39:47.736387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736343 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gdbd\"" Oct 30 20:39:47.736387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736377 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736395 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736412 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736448 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ndwvd\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736477 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736490 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736524 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.736541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736528 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736640 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736689 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736726 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736781 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736812 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.736867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736731 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736871 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736879 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl5jd\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736910 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.736929 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737007 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737007 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737124 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737155 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737173 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737346 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b74p\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737412 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737423 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dctv6\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737458 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kszdw\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737495 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737505 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737550 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5p7gr\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737558 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 20:39:47.737611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737571 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737631 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737725 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737792 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737859 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w89q8\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737879 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.737953 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738055 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g4v6z\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738216 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7chw8\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738242 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5kr7\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738318 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:39:47.738384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738379 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2twjb\"" Oct 30 20:39:47.738815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738428 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cr4sq\"" Oct 30 20:39:47.738815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738542 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:39:47.738815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.738777 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7wk5\"" Oct 30 20:39:47.739852 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.739837 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.740538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.740521 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:39:47.740773 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.740761 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 20:39:47.740853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.740843 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9jnnb\"" Oct 30 20:39:47.741344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.741329 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:39:47.741576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.741564 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 20:39:47.742227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742210 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6ml5f\"" Oct 30 20:39:47.742430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742413 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 20:39:47.742471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742456 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:39:47.742571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742559 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 20:39:47.742607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742573 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 20:39:47.742607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742596 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 20:39:47.742716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.742684 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b2k4x\"" Oct 30 20:39:47.743822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.743809 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:39:47.744079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744064 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:39:47.744174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744162 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:39:47.744500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744486 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-tcxgx\"" Oct 30 20:39:47.744538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744526 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:39:47.744610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744589 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 20:39:47.744610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744608 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qc5gl\"" Oct 30 20:39:47.744817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744644 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jspb5\"" Oct 30 20:39:47.744817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744685 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:39:47.744817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744759 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:39:47.744817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744766 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:39:47.744995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.744908 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a0bik8i2clpn0\"" Oct 30 20:39:47.755357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755338 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.755432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755361 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.755432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755384 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.755432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755408 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.755551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755461 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.755551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755497 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-tmp-dir\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.755551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755514 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.755694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755628 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.755745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755722 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.755793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755760 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3960ed57-c28a-413f-817e-355fa6029130-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.755793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755763 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.755876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755812 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km96v\" (UniqueName: \"kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.755876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755841 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.755876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755842 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.755986 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755964 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.755986 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.755972 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.756078 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756031 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.756078 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756059 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcqv\" (UniqueName: \"kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.756078 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756076 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.756224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756113 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.756224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756202 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.756312 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756255 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.756423 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756391 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.756484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756467 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54acb4d1-67b7-4ac8-9884-11af4f940626-serviceca\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.756532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756511 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-tls\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.756532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756522 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.756638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756573 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.756638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756610 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.756729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756685 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf98h\" (UniqueName: \"kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:47.756778 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756731 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-etc-tuned\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.756778 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756749 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.756882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756833 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.756882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756851 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.756979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756888 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.756979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756929 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-textfile\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.757070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.756978 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gl9h\" (UniqueName: \"kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.757070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757053 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cni-binary-copy\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757131 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.757221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757158 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.757221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757192 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4thr\" (UniqueName: \"kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.757331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757227 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.757331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757262 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757296 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.757331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757328 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757362 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757400 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757407 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-config-volume\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757422 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757455 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757456 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757496 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757525 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx7w\" (UniqueName: \"kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757569 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757610 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757626 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0208ffae-7118-4175-8e46-97ac0592ebfc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.757695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757640 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:47.757881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757753 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9m8\" (UniqueName: \"kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.757881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757785 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.757881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757802 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.757881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757834 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8bd09df0-d50b-4f00-863d-276efe60e695-agent-certs\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:47.757881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757836 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.758047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757926 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.758047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757954 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.758047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.757975 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.758047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758008 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.758047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758021 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1565be2d-9663-42b1-bc86-72e4263fe7c3-iptables-alerter-script\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758055 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758072 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758095 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758116 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758159 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5fw5\" (UniqueName: \"kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758167 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.758250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758219 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758307 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758361 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758367 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-accelerators-collector-config\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758418 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758449 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdjh\" (UniqueName: \"kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758474 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758500 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758528 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758555 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758559 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3960ed57-c28a-413f-817e-355fa6029130-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.758584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758578 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-original-pull-secret\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758580 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758617 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758650 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758701 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758735 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfq4\" (UniqueName: \"kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758744 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758764 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758789 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758795 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8bd09df0-d50b-4f00-863d-276efe60e695-konnectivity-ca\") pod \"konnectivity-agent-ll9wf\" (UID: \"8bd09df0-d50b-4f00-863d-276efe60e695\") " pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758798 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758815 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-metrics-tls\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758835 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758890 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758926 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758946 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758959 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758966 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.758999 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0208ffae-7118-4175-8e46-97ac0592ebfc-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759013 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b22b136-e779-4470-b5ab-b894512f0294-tmp-dir\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759007 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759050 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759072 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8866b225-b24b-43f3-b2a5-960fa1d9f751-cert\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759092 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759115 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759151 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759192 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759211 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f2b5e0-aaed-456d-a023-c2bbf64765dd-metrics-client-ca\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759217 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759231 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47a8078d-c222-4943-af36-4285c0b450cd-tmp\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759249 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759267 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759309 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759326 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759342 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759358 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759407 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-config\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759413 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-ovnkube-script-lib\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759411 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759443 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/26071f56-69e8-4327-a05f-deaf1f8e209a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759447 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759475 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3960ed57-c28a-413f-817e-355fa6029130-data-volume\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759550 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759580 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759582 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb3812a-4ba1-46e5-9433-c9f830ac9629-metrics-certs\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759688 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759719 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93073bfa-1168-4c11-a950-1f75458694d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759708 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759743 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.759782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759788 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnm4\" (UniqueName: \"kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.759974 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760102 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760122 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760140 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760156 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kjl\" (UniqueName: \"kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760170 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26071f56-69e8-4327-a05f-deaf1f8e209a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760248 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760262 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f744cc86-1e11-4932-894f-8610acdcff4e-env-overrides\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760298 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzx65\" (UniqueName: \"kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760309 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-daemon-config\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760316 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760333 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rng4\" (UniqueName: \"kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760349 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760365 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760392 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760413 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760443 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760472 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760494 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760521 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vcm\" (UniqueName: \"kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760550 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760560 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.760789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.760676 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f744cc86-1e11-4932-894f-8610acdcff4e-ovn-node-metrics-cert\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.766946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.766923 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcqv\" (UniqueName: \"kubernetes.io/projected/3960ed57-c28a-413f-817e-355fa6029130-kube-api-access-4rcqv\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.767124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.767109 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf98h\" (UniqueName: \"kubernetes.io/projected/8866b225-b24b-43f3-b2a5-960fa1d9f751-kube-api-access-mf98h\") pod \"ingress-canary-jggzv\" (UID: \"8866b225-b24b-43f3-b2a5-960fa1d9f751\") " pod="openshift-ingress-canary/ingress-canary-jggzv" Oct 30 20:39:47.767349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.767334 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv46\" (UniqueName: \"kubernetes.io/projected/26071f56-69e8-4327-a05f-deaf1f8e209a-kube-api-access-xwv46\") pod \"kube-state-metrics-5dc8d7d89b-v8jvb\" (UID: \"26071f56-69e8-4327-a05f-deaf1f8e209a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v8jvb" Oct 30 20:39:47.772700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.772643 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km96v\" (UniqueName: \"kubernetes.io/projected/0208ffae-7118-4175-8e46-97ac0592ebfc-kube-api-access-km96v\") pod \"openshift-state-metrics-7bd7489957-kwjt8\" (UID: \"0208ffae-7118-4175-8e46-97ac0592ebfc\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-kwjt8" Oct 30 20:39:47.772865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.772760 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx7w\" (UniqueName: \"kubernetes.io/projected/1565be2d-9663-42b1-bc86-72e4263fe7c3-kube-api-access-fwx7w\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.772865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.772857 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4thr\" (UniqueName: \"kubernetes.io/projected/16f2b5e0-aaed-456d-a023-c2bbf64765dd-kube-api-access-c4thr\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.772962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.772905 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gl9h\" (UniqueName: \"kubernetes.io/projected/f744cc86-1e11-4932-894f-8610acdcff4e-kube-api-access-2gl9h\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.775595 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.775573 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5fw5\" (UniqueName: \"kubernetes.io/projected/01f5a5d4-d276-45f8-869e-a41dbd0d445b-kube-api-access-s5fw5\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.777311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777286 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kjl\" (UniqueName: \"kubernetes.io/projected/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kube-api-access-55kjl\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.777500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777453 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzx65\" (UniqueName: \"kubernetes.io/projected/47a8078d-c222-4943-af36-4285c0b450cd-kube-api-access-rzx65\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.777587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777515 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9m8\" (UniqueName: \"kubernetes.io/projected/54acb4d1-67b7-4ac8-9884-11af4f940626-kube-api-access-4j9m8\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.777705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777683 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vnf\" (UniqueName: \"kubernetes.io/projected/a557f314-88c8-4807-aeb5-6e73cf74a44c-kube-api-access-s8vnf\") pod \"network-check-target-ktb85\" (UID: \"a557f314-88c8-4807-aeb5-6e73cf74a44c\") " pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:47.777765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777711 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfq4\" (UniqueName: \"kubernetes.io/projected/0b22b136-e779-4470-b5ab-b894512f0294-kube-api-access-dbfq4\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.777836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777803 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnm4\" (UniqueName: \"kubernetes.io/projected/7bb3812a-4ba1-46e5-9433-c9f830ac9629-kube-api-access-dnnm4\") pod \"network-metrics-daemon-bdndq\" (UID: \"7bb3812a-4ba1-46e5-9433-c9f830ac9629\") " pod="openshift-multus/network-metrics-daemon-bdndq" Oct 30 20:39:47.777905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777886 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vcm\" (UniqueName: \"kubernetes.io/projected/8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4-kube-api-access-92vcm\") pod \"dns-default-m74jp\" (UID: \"8cd103ef-40a7-49a8-a4c7-bc5295ff7cc4\") " pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:47.777983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.777964 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdjh\" (UniqueName: \"kubernetes.io/projected/93073bfa-1168-4c11-a950-1f75458694d1-kube-api-access-csdjh\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.782888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.782868 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rng4\" (UniqueName: \"kubernetes.io/projected/4eca690e-c8c5-4526-bdc4-7cb0a436fc30-kube-api-access-9rng4\") pod \"prometheus-operator-9cd89b74c-6rrvz\" (UID: \"4eca690e-c8c5-4526-bdc4-7cb0a436fc30\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6rrvz" Oct 30 20:39:47.835458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.835425 5855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.835627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.835495 5855 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.837179 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.837161 5855 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:39:47.845918 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.845893 5855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:47.846036 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:47.845939 5855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.846584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.846570 5855 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:47.846635 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:39:47.846604 5855 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.us-east-2.compute.internal" Oct 30 20:39:47.861020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.860977 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861015 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861038 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861065 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861090 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-slash\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861093 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861118 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-systemd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861131 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-bin\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861187 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-socket-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861190 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.861253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861248 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-node-exporter-wtmp\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861276 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861317 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861346 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861353 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-conf-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861373 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861402 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqpw\" (UniqueName: \"kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861426 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861376 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-host\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861477 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861503 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-conf\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861510 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.861639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861619 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861678 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861769 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-sys-fs\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861784 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-tls\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861828 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861829 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-sys\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861877 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-kubelet-config\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861932 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861971 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-netns\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.861973 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-client-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862003 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862015 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862040 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862070 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-cnibin\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862119 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862143 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862154 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-systemd\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862159 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-etc-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862168 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862193 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-node-log\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862202 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862218 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b-dbus\") pod \"global-pull-secret-syncer-wnpsx\" (UID: \"fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b\") " pod="kube-system/global-pull-secret-syncer-wnpsx" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862232 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862258 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862314 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862342 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862366 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862422 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.862416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862429 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-run\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862447 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862527 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-cni-netd\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862678 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ac2e542-da70-4445-a126-d02866a7e2f9-audit-log\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862728 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862758 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862783 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862793 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ac2e542-da70-4445-a126-d02866a7e2f9-secret-metrics-server-client-certs\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862807 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862842 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862858 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b22b136-e779-4470-b5ab-b894512f0294-hosts-file\") pod \"node-resolver-bfbs4\" (UID: \"0b22b136-e779-4470-b5ab-b894512f0294\") " pod="openshift-dns/node-resolver-bfbs4" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862867 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862895 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbt9\" (UniqueName: \"kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862915 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862918 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-device-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862928 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnht\" (UniqueName: \"kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862969 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-system-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.862983 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863014 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863043 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysconfig\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863073 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863101 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863125 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863149 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863172 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863197 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863220 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863246 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863281 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863308 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863332 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863360 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863376 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1565be2d-9663-42b1-bc86-72e4263fe7c3-host-slash\") pod \"iptables-alerter-x874n\" (UID: \"1565be2d-9663-42b1-bc86-72e4263fe7c3\") " pod="openshift-network-operator/iptables-alerter-x874n" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863387 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863426 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863432 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-kubelet\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863421 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863470 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-run-netns\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863476 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863485 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863512 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863517 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-log-socket\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863558 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863576 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3960ed57-c28a-413f-817e-355fa6029130-crio-socket\") pod \"insights-runtime-extractor-wm9wk\" (UID: \"3960ed57-c28a-413f-817e-355fa6029130\") " pod="openshift-insights/insights-runtime-extractor-wm9wk" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863597 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863597 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-lib-modules\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863644 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-root\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863703 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-etc-kubernetes\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863741 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-modprobe-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.863742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863748 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-k8s-cni-cncf-io\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863793 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-registration-dir\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863793 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93073bfa-1168-4c11-a950-1f75458694d1-system-cni-dir\") pod \"multus-additional-cni-plugins-gd9bl\" (UID: \"93073bfa-1168-4c11-a950-1f75458694d1\") " pod="openshift-multus/multus-additional-cni-plugins-gd9bl" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863840 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-systemd-units\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863870 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863895 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863926 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863947 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54acb4d1-67b7-4ac8-9884-11af4f940626-host\") pod \"node-ca-5cmmq\" (UID: \"54acb4d1-67b7-4ac8-9884-11af4f940626\") " pod="openshift-image-registry/node-ca-5cmmq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863646 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-kubernetes\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.863974 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864003 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-socket-dir-parent\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864051 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-var-lib-kubelet\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864048 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1912739c-4112-436d-9000-e5858dce9aeb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864118 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-var-lib-openvswitch\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864000 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864171 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864196 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864221 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864239 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864248 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864256 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ac2e542-da70-4445-a126-d02866a7e2f9-metrics-server-audit-profiles\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864300 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864306 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-bin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864319 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/51cdc667-f01e-435d-8cf8-d7e805e0efd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzrxz\" (UID: \"51cdc667-f01e-435d-8cf8-d7e805e0efd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzrxz" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864373 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f2b5e0-aaed-456d-a023-c2bbf64765dd-sys\") pod \"node-exporter-xx98g\" (UID: \"16f2b5e0-aaed-456d-a023-c2bbf64765dd\") " pod="openshift-monitoring/node-exporter-xx98g" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864410 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864430 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-host-kubelet\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864433 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-hostroot\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864454 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864477 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864481 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47a8078d-c222-4943-af36-4285c0b450cd-etc-sysctl-d\") pod \"tuned-vbqx9\" (UID: \"47a8078d-c222-4943-af36-4285c0b450cd\") " pod="openshift-cluster-node-tuning-operator/tuned-vbqx9" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864513 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-var-lib-cni-multus\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864514 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhg9m\" (UniqueName: \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864524 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-multus-cni-dir\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864546 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-cnibin\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864599 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864628 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864670 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864703 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01f5a5d4-d276-45f8-869e-a41dbd0d445b-host-run-multus-certs\") pod \"multus-65qrq\" (UID: \"01f5a5d4-d276-45f8-869e-a41dbd0d445b\") " pod="openshift-multus/multus-65qrq" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864733 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3039da16-1889-4c78-b2c0-4b612a0cb125-tmp\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:47.865502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.864750 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f744cc86-1e11-4932-894f-8610acdcff4e-run-ovn\") pod \"ovnkube-node-z55gr\" (UID: \"f744cc86-1e11-4932-894f-8610acdcff4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:47.874934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.874903 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqpw\" (UniqueName: \"kubernetes.io/projected/1912739c-4112-436d-9000-e5858dce9aeb-kube-api-access-tfqpw\") pod \"monitoring-plugin-77f4f76594-dck4k\" (UID: \"1912739c-4112-436d-9000-e5858dce9aeb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:47.875669 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.875622 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnht\" (UniqueName: \"kubernetes.io/projected/3039da16-1889-4c78-b2c0-4b612a0cb125-kube-api-access-rmnht\") pod \"downloads-6dd789c8b7-gzv86\" (UID: \"3039da16-1889-4c78-b2c0-4b612a0cb125\") " pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:39:47.875798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.875759 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbt9\" (UniqueName: \"kubernetes.io/projected/3ac2e542-da70-4445-a126-d02866a7e2f9-kube-api-access-hkbt9\") pod \"metrics-server-656d8b8c49-pmmgw\" (UID: \"3ac2e542-da70-4445-a126-d02866a7e2f9\") " pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:39:47.965950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.965849 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.965950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.965896 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhg9m\" (UniqueName: \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.965950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.965925 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.965950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.965952 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.965990 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966023 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966043 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966451 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966424 5855 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:39:47.966531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966511 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966726 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966729 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.966862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.966810 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.968738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.968713 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.968868 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.968851 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:47.978594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:47.978571 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhg9m\" (UniqueName: \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\") pod \"console-7f69955664-z6kgn\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:48.040422 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:48.040378 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:48.040596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:48.040579 5855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:39:48.316014 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:48.315978 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:39:48.319450 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:48.319419 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9960ee5_81b6_4b66_bc69_fda7e923e0d5.slice/crio-9d99d40b78308104a82b191360d14e0a5d55502f27bc3f06674b99e23ab07b08 WatchSource:0}: Error finding container 9d99d40b78308104a82b191360d14e0a5d55502f27bc3f06674b99e23ab07b08: Status 404 returned error can't find the container with id 9d99d40b78308104a82b191360d14e0a5d55502f27bc3f06674b99e23ab07b08 Oct 30 20:39:48.809895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:48.809863 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:39:48.846088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:48.843330 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f69955664-z6kgn" event={"ID":"f9960ee5-81b6-4b66-bc69-fda7e923e0d5","Type":"ContainerStarted","Data":"9d99d40b78308104a82b191360d14e0a5d55502f27bc3f06674b99e23ab07b08"} Oct 30 20:39:49.382356 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.382313 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:39:49.386066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.386034 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.409500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.409433 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 20:39:49.410928 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.410903 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:39:49.481509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481475 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5ps\" (UniqueName: \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481517 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481538 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481578 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481677 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481706 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.481750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481744 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.482034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.481772 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583036 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583084 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583119 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583145 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583173 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5ps\" (UniqueName: \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.583330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583201 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.584634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583693 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.584634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.583755 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.584634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.584553 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.584858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.584703 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.584858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.584707 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.585292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.585268 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.585898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.585875 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.586221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.586200 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.586527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.586507 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.604886 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.604853 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5ps\" (UniqueName: \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\") pod \"console-575cf5df8d-wsjc4\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.702242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.702156 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:49.990319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:49.990199 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:39:49.994618 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:39:49.994589 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef38178_3e7b_4e9a_a8f3_4f386ffa52e2.slice/crio-3ea21e7b9cf44fdbfdbe92d75c334a82cd86c77aa366fe5208baa64634e389c8 WatchSource:0}: Error finding container 3ea21e7b9cf44fdbfdbe92d75c334a82cd86c77aa366fe5208baa64634e389c8: Status 404 returned error can't find the container with id 3ea21e7b9cf44fdbfdbe92d75c334a82cd86c77aa366fe5208baa64634e389c8 Oct 30 20:39:50.751017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:50.750981 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m74jp" Oct 30 20:39:50.865096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:50.865045 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cf5df8d-wsjc4" event={"ID":"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2","Type":"ContainerStarted","Data":"3ea21e7b9cf44fdbfdbe92d75c334a82cd86c77aa366fe5208baa64634e389c8"} Oct 30 20:39:51.138571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:51.137839 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:51.144072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:51.144040 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" Oct 30 20:39:52.879700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:52.879626 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cf5df8d-wsjc4" event={"ID":"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2","Type":"ContainerStarted","Data":"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6"} Oct 30 20:39:52.881686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:52.881641 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f69955664-z6kgn" event={"ID":"f9960ee5-81b6-4b66-bc69-fda7e923e0d5","Type":"ContainerStarted","Data":"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390"} Oct 30 20:39:52.909490 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:52.909427 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-575cf5df8d-wsjc4" podStartSLOduration=1.9144030170000002 podStartE2EDuration="3.909412944s" totalImagesPullingTime="1.995009927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:52.907285697 +0000 UTC m=+6.265998444" watchObservedRunningTime="2026-10-30 20:39:52.909412944 +0000 UTC m=+6.268125686" Oct 30 20:39:52.938075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:52.938008 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f69955664-z6kgn" podStartSLOduration=3.311427246 podStartE2EDuration="6.937982532s" totalImagesPullingTime="3.626555286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:52.930292339 +0000 UTC m=+6.289005085" watchObservedRunningTime="2026-10-30 20:39:52.937982532 +0000 UTC m=+6.296695279" Oct 30 20:39:53.017178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:53.017136 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:54.587114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:54.587029 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:54.587544 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:54.587247 5855 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:39:56.681645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:56.681589 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:39:57.492922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.492896 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z55gr" Oct 30 20:39:57.634055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.634023 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:57.639786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.639755 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:57.901817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.901788 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:57.907279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.907253 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:39:57.960360 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:39:57.960325 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:40:00.856239 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:00.856185 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:40:00.940698 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:40:00.940634 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:40:01.168599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:01.168515 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ll9wf" Oct 30 20:40:03.090381 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:03.090338 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:40:04.934275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:04.934239 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-gzv86" event={"ID":"3039da16-1889-4c78-b2c0-4b612a0cb125","Type":"ContainerStarted","Data":"5486dcd970f45956be2086f817dbbeeef9a0797f9ca254c12a81c4b36f0cd5b1"} Oct 30 20:40:04.934763 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:04.934519 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:40:04.935962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:04.935941 5855 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-gzv86 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" start-of-body= Oct 30 20:40:04.936033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:04.935984 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-gzv86" podUID="3039da16-1889-4c78-b2c0-4b612a0cb125" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" Oct 30 20:40:05.951403 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:05.951348 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-gzv86" Oct 30 20:40:18.846205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:18.846167 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ktb85" Oct 30 20:40:20.861066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:20.861040 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:40:20.864993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:20.864970 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" Oct 30 20:40:22.983326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:22.983274 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f69955664-z6kgn" podUID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" containerName="console" containerID="cri-o://7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390" gracePeriod=15 Oct 30 20:40:23.264264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.264114 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:40:23.266597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.266554 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.266890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.266649 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.266890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.266880 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.267016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.266938 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\" (UniqueName: \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.267016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.266988 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.267110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267025 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.267163 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267128 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\") pod \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\" (UID: \"f9960ee5-81b6-4b66-bc69-fda7e923e0d5\") " Oct 30 20:40:23.267309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267247 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:23.267421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267258 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:23.267421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267255 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:23.267720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267691 5855 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-oauth-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.267720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267724 5855 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-service-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.267975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267737 5855 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-tmp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.267975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.267795 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:23.270760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.270701 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:23.270760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.270729 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:23.270760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.270708 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m" pod "f9960ee5-81b6-4b66-bc69-fda7e923e0d5" (UID: "f9960ee5-81b6-4b66-bc69-fda7e923e0d5"). InnerVolumeSpecName "kube-api-access-hhg9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:23.368587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.368548 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhg9m\" (UniqueName: \"kubernetes.io/projected/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-kube-api-access-hhg9m\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.368587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.368581 5855 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-oauth-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.368587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.368593 5855 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:23.368870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:23.368604 5855 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9960ee5-81b6-4b66-bc69-fda7e923e0d5-console-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:24.015499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.015465 5855 generic.go:356] "Generic (PLEG): container finished" podID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" containerID="7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390" exitCode=0 Oct 30 20:40:24.015955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.015509 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f69955664-z6kgn" event={"ID":"f9960ee5-81b6-4b66-bc69-fda7e923e0d5","Type":"ContainerDied","Data":"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390"} Oct 30 20:40:24.015955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.015513 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f69955664-z6kgn" Oct 30 20:40:24.015955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.015548 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f69955664-z6kgn" event={"ID":"f9960ee5-81b6-4b66-bc69-fda7e923e0d5","Type":"ContainerDied","Data":"9d99d40b78308104a82b191360d14e0a5d55502f27bc3f06674b99e23ab07b08"} Oct 30 20:40:24.015955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.015568 5855 scope.go:122] "RemoveContainer" containerID="7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390" Oct 30 20:40:24.025329 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.025312 5855 scope.go:122] "RemoveContainer" containerID="7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390" Oct 30 20:40:24.025603 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:40:24.025580 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390\": container with ID starting with 7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390 not found: ID does not exist" containerID="7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390" Oct 30 20:40:24.025721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.025610 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390"} err="failed to get container status \"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390\": rpc error: code = NotFound desc = could not find container \"7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390\": container with ID starting with 7f3df11756e778a15cd6e835dc78b0ab9a03e752a5a30ebe49a9bc7204e15390 not found: ID does not exist" Oct 30 20:40:24.036140 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.036120 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:40:24.040127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.040106 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f69955664-z6kgn"] Oct 30 20:40:24.776830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:24.776790 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" path="/var/lib/kubelet/pods/f9960ee5-81b6-4b66-bc69-fda7e923e0d5/volumes" Oct 30 20:40:30.924541 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:40:30.924504 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:40:44.784696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.784639 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7cc949bd99-npw2l"] Oct 30 20:40:44.785148 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.784952 5855 cpu_manager.go:414] "containerMap: removing container" podUID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" containerName="console" Oct 30 20:40:44.785148 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.784969 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" containerName="console" Oct 30 20:40:44.785148 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.785040 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9960ee5-81b6-4b66-bc69-fda7e923e0d5" containerName="console" Oct 30 20:40:44.821526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.821492 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cc949bd99-npw2l"] Oct 30 20:40:44.821713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.821617 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.832245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.832217 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 20:40:44.832484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.832438 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 20:40:44.832484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.832435 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 20:40:44.832484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.832486 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 20:40:44.832619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.832491 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bg9hs\"" Oct 30 20:40:44.907044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.907007 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 20:40:44.909458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.909407 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 20:40:44.931755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.931711 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.931755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.931757 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-metrics-client-ca\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.931957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.931876 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-federate-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.931992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.931958 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.932025 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.931990 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.932025 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.932014 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdxn\" (UniqueName: \"kubernetes.io/projected/211b1aff-4f75-44a0-ac18-19e213b6519f-kube-api-access-gwdxn\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.932131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.932112 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-serving-certs-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:44.932165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:44.932149 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033361 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033323 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033370 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033392 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdxn\" (UniqueName: \"kubernetes.io/projected/211b1aff-4f75-44a0-ac18-19e213b6519f-kube-api-access-gwdxn\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033428 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-serving-certs-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033451 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033482 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033504 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-metrics-client-ca\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.033573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.033545 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-federate-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.034429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.034400 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-metrics-client-ca\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.034429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.034417 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-serving-certs-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.034770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.034725 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.036326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.036296 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-telemeter-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.036510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.036487 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-federate-client-tls\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.036713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.036693 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.036787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.036727 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/211b1aff-4f75-44a0-ac18-19e213b6519f-secret-telemeter-client\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.046822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.046787 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdxn\" (UniqueName: \"kubernetes.io/projected/211b1aff-4f75-44a0-ac18-19e213b6519f-kube-api-access-gwdxn\") pod \"telemeter-client-7cc949bd99-npw2l\" (UID: \"211b1aff-4f75-44a0-ac18-19e213b6519f\") " pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.173398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.173358 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" Oct 30 20:40:45.495918 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:45.495887 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7cc949bd99-npw2l"] Oct 30 20:40:45.499966 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:40:45.499929 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211b1aff_4f75_44a0_ac18_19e213b6519f.slice/crio-4aa82c752cd238f55e5135a893a90287dd0d21dcefb83dd2bde51949d2111674 WatchSource:0}: Error finding container 4aa82c752cd238f55e5135a893a90287dd0d21dcefb83dd2bde51949d2111674: Status 404 returned error can't find the container with id 4aa82c752cd238f55e5135a893a90287dd0d21dcefb83dd2bde51949d2111674 Oct 30 20:40:46.098173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:46.098122 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" event={"ID":"211b1aff-4f75-44a0-ac18-19e213b6519f","Type":"ContainerStarted","Data":"4aa82c752cd238f55e5135a893a90287dd0d21dcefb83dd2bde51949d2111674"} Oct 30 20:40:48.111010 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:48.110973 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" event={"ID":"211b1aff-4f75-44a0-ac18-19e213b6519f","Type":"ContainerStarted","Data":"72d90151e37883713cb185df2d992a900b2ac474297558ad241980b37e8d9b0c"} Oct 30 20:40:50.120302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.120262 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" event={"ID":"211b1aff-4f75-44a0-ac18-19e213b6519f","Type":"ContainerStarted","Data":"72d564ffd27bbcb71965590f5e672f0bf76376678c3ee9ead2bad9d9cbc590c9"} Oct 30 20:40:50.120730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.120330 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" event={"ID":"211b1aff-4f75-44a0-ac18-19e213b6519f","Type":"ContainerStarted","Data":"8c6c966d1f7dbee925f4d16b26e7a5dc058949ec0c9e47a12ff5cd738a60725c"} Oct 30 20:40:50.154981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.154931 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7cc949bd99-npw2l" podStartSLOduration=2.478758173 podStartE2EDuration="6.154918388s" totalImagesPullingTime="3.676160215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:50.153380591 +0000 UTC m=+63.512093337" watchObservedRunningTime="2026-10-30 20:40:50.154918388 +0000 UTC m=+63.513631133" Oct 30 20:40:50.252639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.252595 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:40:50.254650 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.254634 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.267087 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.267064 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:40:50.376426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376326 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376366 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376390 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376409 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376437 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376457 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376553 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.376765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.376615 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbc6\" (UniqueName: \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477212 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbc6\" (UniqueName: \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477278 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477305 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477338 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477365 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477385 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477409 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.477765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.477507 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.478115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.478062 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.478251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.478235 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.478311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.478266 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.478311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.478263 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.478534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.478513 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.480590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.480574 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.480739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.480720 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.490153 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.490130 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbc6\" (UniqueName: \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\") pod \"console-c54d597b4-qklk8\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.565130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.565091 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:40:50.838089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:50.838053 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:40:50.842131 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:40:50.842094 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36ed184_9d54_4d61_ad42_44cab87c16ee.slice/crio-ed4ca39578814c3c182fab787c1acaf6b17c4ab97934b41d92877395478c6ab5 WatchSource:0}: Error finding container ed4ca39578814c3c182fab787c1acaf6b17c4ab97934b41d92877395478c6ab5: Status 404 returned error can't find the container with id ed4ca39578814c3c182fab787c1acaf6b17c4ab97934b41d92877395478c6ab5 Oct 30 20:40:51.127429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:51.127303 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54d597b4-qklk8" event={"ID":"a36ed184-9d54-4d61-ad42-44cab87c16ee","Type":"ContainerStarted","Data":"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b"} Oct 30 20:40:51.127429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:40:51.127389 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54d597b4-qklk8" event={"ID":"a36ed184-9d54-4d61-ad42-44cab87c16ee","Type":"ContainerStarted","Data":"ed4ca39578814c3c182fab787c1acaf6b17c4ab97934b41d92877395478c6ab5"} Oct 30 20:41:00.565973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:00.565928 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:41:00.566576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:00.566008 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:41:00.570686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:00.570649 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:41:00.592797 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:00.592724 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c54d597b4-qklk8" podStartSLOduration=10.592691241 podStartE2EDuration="10.592691241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:51.154586767 +0000 UTC m=+64.513299514" watchObservedRunningTime="2026-10-30 20:41:00.592691241 +0000 UTC m=+73.951403991" Oct 30 20:41:00.924420 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:41:00.924336 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:41:01.164534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:01.164489 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:41:01.217104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:01.217017 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:41:26.242289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.242187 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-575cf5df8d-wsjc4" podUID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" containerName="console" containerID="cri-o://a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6" gracePeriod=15 Oct 30 20:41:26.493116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.493053 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:41:26.673106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673069 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673120 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\" (UniqueName: \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673144 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673161 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673343 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673393 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673437 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673458 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:26.673717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673481 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\") pod \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\" (UID: \"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2\") " Oct 30 20:41:26.673717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673517 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:26.673717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673623 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:26.673859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673799 5855 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-service-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.673859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673820 5855 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.673859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673842 5855 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-tmp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.674170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.673869 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:26.674231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.674167 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:41:26.675555 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.675526 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:26.675677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.675555 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:41:26.675677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.675624 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps" pod "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" (UID: "2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2"). InnerVolumeSpecName "kube-api-access-8w5ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:26.774373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.774287 5855 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-oauth-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.774373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.774312 5855 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-oauth-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.774373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.774326 5855 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-trusted-ca-bundle\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.774373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.774341 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w5ps\" (UniqueName: \"kubernetes.io/projected/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-kube-api-access-8w5ps\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:26.774373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:26.774355 5855 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2-console-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:27.247428 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.247385 5855 generic.go:356] "Generic (PLEG): container finished" podID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" containerID="a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6" exitCode=0 Oct 30 20:41:27.247957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.247464 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cf5df8d-wsjc4" event={"ID":"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2","Type":"ContainerDied","Data":"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6"} Oct 30 20:41:27.247957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.247554 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575cf5df8d-wsjc4" event={"ID":"2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2","Type":"ContainerDied","Data":"3ea21e7b9cf44fdbfdbe92d75c334a82cd86c77aa366fe5208baa64634e389c8"} Oct 30 20:41:27.247957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.247480 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575cf5df8d-wsjc4" Oct 30 20:41:27.247957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.247575 5855 scope.go:122] "RemoveContainer" containerID="a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6" Oct 30 20:41:27.259150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.259134 5855 scope.go:122] "RemoveContainer" containerID="a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6" Oct 30 20:41:27.259410 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:41:27.259392 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6\": container with ID starting with a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6 not found: ID does not exist" containerID="a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6" Oct 30 20:41:27.259493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.259432 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6"} err="failed to get container status \"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6\": rpc error: code = NotFound desc = could not find container \"a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6\": container with ID starting with a7158448fde66031480c1e77c7a3e0965458a952e7917ab0690340a0faa961d6 not found: ID does not exist" Oct 30 20:41:27.265503 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.265480 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:41:27.269966 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:27.269942 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575cf5df8d-wsjc4"] Oct 30 20:41:28.776810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:41:28.776779 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" path="/var/lib/kubelet/pods/2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2/volumes" Oct 30 20:41:30.924444 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:41:30.924411 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:42:00.928429 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:42:00.928386 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:42:02.075836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.075801 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c94597f6-22wt5"] Oct 30 20:42:02.076220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.076068 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" containerName="console" Oct 30 20:42:02.076220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.076078 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" containerName="console" Oct 30 20:42:02.076220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.076145 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef38178-3e7b-4e9a-a8f3-4f386ffa52e2" containerName="console" Oct 30 20:42:02.078958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.078939 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.091521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.091481 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c94597f6-22wt5"] Oct 30 20:42:02.127891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.127859 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsd7k\" (UniqueName: \"kubernetes.io/projected/0e5d4074-6876-4630-973d-78c24ef0c7e6-kube-api-access-bsd7k\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.127891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.127894 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-trusted-ca-bundle\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.127938 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.128022 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-service-ca\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.128062 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-oauth-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.128079 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-oauth-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.128148 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5d4074-6876-4630-973d-78c24ef0c7e6-tmp\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.128267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.128181 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229070 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229132 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-service-ca\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229170 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-oauth-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229196 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-oauth-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229239 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5d4074-6876-4630-973d-78c24ef0c7e6-tmp\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229268 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229306 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsd7k\" (UniqueName: \"kubernetes.io/projected/0e5d4074-6876-4630-973d-78c24ef0c7e6-kube-api-access-bsd7k\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229333 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-trusted-ca-bundle\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.229939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.229876 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e5d4074-6876-4630-973d-78c24ef0c7e6-tmp\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.230253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.230232 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-service-ca\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.230329 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.230311 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-trusted-ca-bundle\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.230460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.230435 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-oauth-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.230532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.230508 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.232027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.232009 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-serving-cert\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.232131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.232115 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e5d4074-6876-4630-973d-78c24ef0c7e6-console-oauth-config\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.242311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.242284 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsd7k\" (UniqueName: \"kubernetes.io/projected/0e5d4074-6876-4630-973d-78c24ef0c7e6-kube-api-access-bsd7k\") pod \"console-69c94597f6-22wt5\" (UID: \"0e5d4074-6876-4630-973d-78c24ef0c7e6\") " pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.391148 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.391059 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:02.664910 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:02.664838 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c94597f6-22wt5"] Oct 30 20:42:02.668067 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:42:02.668035 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5d4074_6876_4630_973d_78c24ef0c7e6.slice/crio-26afe08e729618a198512f4f999bdfeffe6f8ffde2874f7ed2452bcf10964c43 WatchSource:0}: Error finding container 26afe08e729618a198512f4f999bdfeffe6f8ffde2874f7ed2452bcf10964c43: Status 404 returned error can't find the container with id 26afe08e729618a198512f4f999bdfeffe6f8ffde2874f7ed2452bcf10964c43 Oct 30 20:42:03.374242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:03.374203 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c94597f6-22wt5" event={"ID":"0e5d4074-6876-4630-973d-78c24ef0c7e6","Type":"ContainerStarted","Data":"3ecb380632db4e8d5fcd503a727a9a692f47631e537ed79ecacdca9aac784b81"} Oct 30 20:42:03.374242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:03.374252 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c94597f6-22wt5" event={"ID":"0e5d4074-6876-4630-973d-78c24ef0c7e6","Type":"ContainerStarted","Data":"26afe08e729618a198512f4f999bdfeffe6f8ffde2874f7ed2452bcf10964c43"} Oct 30 20:42:03.396612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:03.396549 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69c94597f6-22wt5" podStartSLOduration=1.396529882 podStartE2EDuration="1.396529882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:03.396279778 +0000 UTC m=+136.754992525" watchObservedRunningTime="2026-10-30 20:42:03.396529882 +0000 UTC m=+136.755242628" Oct 30 20:42:12.392061 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:12.392006 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:12.392603 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:12.392092 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:12.396998 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:12.396972 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:12.409144 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:12.409119 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c94597f6-22wt5" Oct 30 20:42:12.470294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:12.470261 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:42:30.922853 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:42:30.922814 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:42:37.491108 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.491048 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c54d597b4-qklk8" podUID="a36ed184-9d54-4d61-ad42-44cab87c16ee" containerName="console" containerID="cri-o://83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b" gracePeriod=15 Oct 30 20:42:37.735136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.735112 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:42:37.822264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822233 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822280 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822310 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822339 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822378 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822421 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822446 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\" (UniqueName: \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822487 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\") pod \"a36ed184-9d54-4d61-ad42-44cab87c16ee\" (UID: \"a36ed184-9d54-4d61-ad42-44cab87c16ee\") " Oct 30 20:42:37.822807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822754 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:42:37.822901 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822807 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:37.822901 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822875 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:37.822990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.822963 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:37.823037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.823012 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:42:37.824912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.824880 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:37.825001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.824974 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "kube-api-access-wjbc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:42:37.825567 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.825545 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config" pod "a36ed184-9d54-4d61-ad42-44cab87c16ee" (UID: "a36ed184-9d54-4d61-ad42-44cab87c16ee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:42:37.923718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923640 5855 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-service-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.923718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923712 5855 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-oauth-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.923718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923722 5855 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a36ed184-9d54-4d61-ad42-44cab87c16ee-tmp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.923718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923732 5855 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.923718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923740 5855 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-console-config\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.924036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923749 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbc6\" (UniqueName: \"kubernetes.io/projected/a36ed184-9d54-4d61-ad42-44cab87c16ee-kube-api-access-wjbc6\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.924036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923759 5855 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-trusted-ca-bundle\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:37.924036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:37.923768 5855 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a36ed184-9d54-4d61-ad42-44cab87c16ee-oauth-serving-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:42:38.497501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.497465 5855 generic.go:356] "Generic (PLEG): container finished" podID="a36ed184-9d54-4d61-ad42-44cab87c16ee" containerID="83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b" exitCode=0 Oct 30 20:42:38.497501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.497504 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54d597b4-qklk8" Oct 30 20:42:38.498072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.497551 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54d597b4-qklk8" event={"ID":"a36ed184-9d54-4d61-ad42-44cab87c16ee","Type":"ContainerDied","Data":"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b"} Oct 30 20:42:38.498072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.497601 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54d597b4-qklk8" event={"ID":"a36ed184-9d54-4d61-ad42-44cab87c16ee","Type":"ContainerDied","Data":"ed4ca39578814c3c182fab787c1acaf6b17c4ab97934b41d92877395478c6ab5"} Oct 30 20:42:38.498072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.497630 5855 scope.go:122] "RemoveContainer" containerID="83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b" Oct 30 20:42:38.510646 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.510627 5855 scope.go:122] "RemoveContainer" containerID="83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b" Oct 30 20:42:38.511014 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:42:38.510994 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b\": container with ID starting with 83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b not found: ID does not exist" containerID="83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b" Oct 30 20:42:38.511060 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.511025 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b"} err="failed to get container status \"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b\": rpc error: code = NotFound desc = could not find container \"83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b\": container with ID starting with 83e30192b14c5c0194d4bb37569472b0ea3ac844eaee5fb7060f220dcd63412b not found: ID does not exist" Oct 30 20:42:38.522076 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.522043 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:42:38.526118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.526096 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c54d597b4-qklk8"] Oct 30 20:42:38.776745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:42:38.776674 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36ed184-9d54-4d61-ad42-44cab87c16ee" path="/var/lib/kubelet/pods/a36ed184-9d54-4d61-ad42-44cab87c16ee/volumes" Oct 30 20:43:00.924619 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:43:00.924539 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:43:30.923457 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:43:30.923422 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:43:42.902821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.902785 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg"] Oct 30 20:43:42.905236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.903103 5855 cpu_manager.go:414] "containerMap: removing container" podUID="a36ed184-9d54-4d61-ad42-44cab87c16ee" containerName="console" Oct 30 20:43:42.905236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.903114 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36ed184-9d54-4d61-ad42-44cab87c16ee" containerName="console" Oct 30 20:43:42.905236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.903167 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36ed184-9d54-4d61-ad42-44cab87c16ee" containerName="console" Oct 30 20:43:42.906224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.906205 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:42.912828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.912794 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-5qxr8\"" Oct 30 20:43:42.912828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.912811 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 20:43:42.912828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.912812 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 20:43:42.915848 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:42.915821 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg"] Oct 30 20:43:43.028101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.028068 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jp8\" (UniqueName: \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.028300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.028109 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.028300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.028206 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.035276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.035248 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg"] Oct 30 20:43:43.038695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.038676 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.043007 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.042985 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xcxhd\"" Oct 30 20:43:43.047649 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.047625 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg"] Oct 30 20:43:43.129110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129064 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.129287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129133 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.129287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129218 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jp8\" (UniqueName: \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.129287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129256 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.129287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129284 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9nn\" (UniqueName: \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.129419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129303 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.129544 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129525 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.129589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.129575 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.142889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.142857 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jp8\" (UniqueName: \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.222984 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.222880 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:43:43.230125 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.230099 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.230210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.230192 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.230266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.230251 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9nn\" (UniqueName: \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.230460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.230441 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.230527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.230509 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.243404 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.243374 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9nn\" (UniqueName: \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.356430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.356389 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:43:43.401752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.401719 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:43:43.405376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.405341 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.417952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.417929 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:43:43.418150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.417985 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:43:43.418150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.418071 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:43:43.418296 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.418168 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:43:43.422764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.422486 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:43:43.422764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.422615 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n56bm\"" Oct 30 20:43:43.422764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.422643 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:43:43.422764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.422738 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:43:43.513576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.513485 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg"] Oct 30 20:43:43.516940 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:43:43.516904 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b44f6d_d754_45da_8359_599eefca6429.slice/crio-743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2 WatchSource:0}: Error finding container 743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2: Status 404 returned error can't find the container with id 743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2 Oct 30 20:43:43.533362 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533330 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533375 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533399 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533482 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533617 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533562 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533617 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533608 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnnd\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.533703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.533631 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.632453 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.632423 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg"] Oct 30 20:43:43.634613 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:43:43.634587 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2337ed82_622d_42ea_982b_6b455f3ea15b.slice/crio-fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c WatchSource:0}: Error finding container fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c: Status 404 returned error can't find the container with id fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c Oct 30 20:43:43.634613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634595 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634671 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnnd\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634710 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634743 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634799 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634837 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.634975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.634876 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.635705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.635510 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.637408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.637389 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.637738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.637712 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.637819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.637746 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.637819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.637756 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.647276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.647249 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.647351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.647302 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnnd\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\") pod \"istiod-openshift-gateway-7876bfcb74-kpwft\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.723617 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.723578 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:43.734553 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.734523 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerStarted","Data":"fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c"} Oct 30 20:43:43.735714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:43.735693 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" event={"ID":"87b44f6d-d754-45da-8359-599eefca6429","Type":"ContainerStarted","Data":"743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2"} Oct 30 20:43:44.005496 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:44.005460 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:43:44.009559 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:43:44.009524 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fd6b93_5ccc_45ed_8ddd_e79e765048da.slice/crio-4b0178a6294745c78b5cdba7f7001ddf33420470bc3fd8fdc3d2702b1564b0f8 WatchSource:0}: Error finding container 4b0178a6294745c78b5cdba7f7001ddf33420470bc3fd8fdc3d2702b1564b0f8: Status 404 returned error can't find the container with id 4b0178a6294745c78b5cdba7f7001ddf33420470bc3fd8fdc3d2702b1564b0f8 Oct 30 20:43:44.742346 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:44.742302 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" event={"ID":"b4fd6b93-5ccc-45ed-8ddd-e79e765048da","Type":"ContainerStarted","Data":"4b0178a6294745c78b5cdba7f7001ddf33420470bc3fd8fdc3d2702b1564b0f8"} Oct 30 20:43:47.848497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:47.848459 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:47.848944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:47.848541 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:43:48.775429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:48.775386 5855 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-kpwft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:43:48.775627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:48.775441 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:43:48.778700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:48.778646 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" event={"ID":"b4fd6b93-5ccc-45ed-8ddd-e79e765048da","Type":"ContainerStarted","Data":"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7"} Oct 30 20:43:48.778872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:48.778731 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:48.801407 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:48.801340 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" podStartSLOduration=1.965668483 podStartE2EDuration="5.801322737s" totalImagesPullingTime="3.835654254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:48.799599603 +0000 UTC m=+242.158312351" watchObservedRunningTime="2026-10-30 20:43:48.801322737 +0000 UTC m=+242.160035493" Oct 30 20:43:49.779264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:49.778743 5855 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-kpwft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:43:49.779264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:49.778809 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:43:50.783875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:50.783845 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerStarted","Data":"ed5ef4c8c6dce39d4c53ee6d7d7878cc56b16f6ec67cc5e1a133b5f2b1cd6d2a"} Oct 30 20:43:50.785387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:50.785357 5855 generic.go:356] "Generic (PLEG): container finished" podID="87b44f6d-d754-45da-8359-599eefca6429" containerID="9ab0da0cd6be5f3416a54366e4ae3ac6aeef58a0864958926d0e0ec88011a379" exitCode=0 Oct 30 20:43:50.785500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:50.785405 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" event={"ID":"87b44f6d-d754-45da-8359-599eefca6429","Type":"ContainerDied","Data":"9ab0da0cd6be5f3416a54366e4ae3ac6aeef58a0864958926d0e0ec88011a379"} Oct 30 20:43:51.791226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:51.791193 5855 generic.go:356] "Generic (PLEG): container finished" podID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerID="ed5ef4c8c6dce39d4c53ee6d7d7878cc56b16f6ec67cc5e1a133b5f2b1cd6d2a" exitCode=0 Oct 30 20:43:51.791643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:51.791241 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerDied","Data":"ed5ef4c8c6dce39d4c53ee6d7d7878cc56b16f6ec67cc5e1a133b5f2b1cd6d2a"} Oct 30 20:43:52.779450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:52.779381 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:43:52.796703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:52.796644 5855 generic.go:356] "Generic (PLEG): container finished" podID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerID="efa54e10543b0856c0a00ff02b0b63a38dc05d4363b584a14f748939a0cd5573" exitCode=0 Oct 30 20:43:52.796703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:52.796688 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerDied","Data":"efa54e10543b0856c0a00ff02b0b63a38dc05d4363b584a14f748939a0cd5573"} Oct 30 20:43:53.803650 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:53.803611 5855 generic.go:356] "Generic (PLEG): container finished" podID="87b44f6d-d754-45da-8359-599eefca6429" containerID="2a71e82e78194c617b11d4b9cf4b9b59592e5f9f3fa62b51455787cf396e5b76" exitCode=0 Oct 30 20:43:53.804181 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:53.803708 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" event={"ID":"87b44f6d-d754-45da-8359-599eefca6429","Type":"ContainerDied","Data":"2a71e82e78194c617b11d4b9cf4b9b59592e5f9f3fa62b51455787cf396e5b76"} Oct 30 20:43:59.835254 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:59.835166 5855 generic.go:356] "Generic (PLEG): container finished" podID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerID="53b1e8f4635de3f99cbc68ae25fb53f2017bd0357bc8771a77d493d8de488818" exitCode=0 Oct 30 20:43:59.835723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:43:59.835280 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerDied","Data":"53b1e8f4635de3f99cbc68ae25fb53f2017bd0357bc8771a77d493d8de488818"} Oct 30 20:44:00.841846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:00.841799 5855 generic.go:356] "Generic (PLEG): container finished" podID="87b44f6d-d754-45da-8359-599eefca6429" containerID="23615578d8fa7fe7fc8e085c6aeb3e8fa404daca215b3ee771c02e689979feac" exitCode=0 Oct 30 20:44:00.842294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:00.841881 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" event={"ID":"87b44f6d-d754-45da-8359-599eefca6429","Type":"ContainerDied","Data":"23615578d8fa7fe7fc8e085c6aeb3e8fa404daca215b3ee771c02e689979feac"} Oct 30 20:44:00.925351 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:44:00.925315 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:44:00.967334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:00.967311 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:44:01.097899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.097810 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\" (UniqueName: \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\") pod \"2337ed82-622d-42ea-982b-6b455f3ea15b\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " Oct 30 20:44:01.097899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.097890 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\") pod \"2337ed82-622d-42ea-982b-6b455f3ea15b\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " Oct 30 20:44:01.098079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.097981 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\") pod \"2337ed82-622d-42ea-982b-6b455f3ea15b\" (UID: \"2337ed82-622d-42ea-982b-6b455f3ea15b\") " Oct 30 20:44:01.098520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.098486 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle" pod "2337ed82-622d-42ea-982b-6b455f3ea15b" (UID: "2337ed82-622d-42ea-982b-6b455f3ea15b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:01.100281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.100249 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn" pod "2337ed82-622d-42ea-982b-6b455f3ea15b" (UID: "2337ed82-622d-42ea-982b-6b455f3ea15b"). InnerVolumeSpecName "kube-api-access-2z9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:01.102269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.102247 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util" pod "2337ed82-622d-42ea-982b-6b455f3ea15b" (UID: "2337ed82-622d-42ea-982b-6b455f3ea15b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:01.199269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.199201 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9nn\" (UniqueName: \"kubernetes.io/projected/2337ed82-622d-42ea-982b-6b455f3ea15b-kube-api-access-2z9nn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:01.199269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.199258 5855 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-util\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:01.199269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.199267 5855 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2337ed82-622d-42ea-982b-6b455f3ea15b-bundle\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:01.847529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.847475 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" event={"ID":"2337ed82-622d-42ea-982b-6b455f3ea15b","Type":"ContainerDied","Data":"fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c"} Oct 30 20:44:01.847529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.847511 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qblbg" Oct 30 20:44:01.848006 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.847546 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc5e507a5f437720d60a3f8b5288739884bde3766ba6e167f7a82dd5897bc78c" Oct 30 20:44:01.975626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:01.975598 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:44:02.105889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.105802 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\") pod \"87b44f6d-d754-45da-8359-599eefca6429\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " Oct 30 20:44:02.106033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.105899 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\") pod \"87b44f6d-d754-45da-8359-599eefca6429\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " Oct 30 20:44:02.106033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.105942 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\" (UniqueName: \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\") pod \"87b44f6d-d754-45da-8359-599eefca6429\" (UID: \"87b44f6d-d754-45da-8359-599eefca6429\") " Oct 30 20:44:02.106265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.106241 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle" pod "87b44f6d-d754-45da-8359-599eefca6429" (UID: "87b44f6d-d754-45da-8359-599eefca6429"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:02.108242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.108205 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8" pod "87b44f6d-d754-45da-8359-599eefca6429" (UID: "87b44f6d-d754-45da-8359-599eefca6429"). InnerVolumeSpecName "kube-api-access-22jp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:02.110383 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.110358 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util" pod "87b44f6d-d754-45da-8359-599eefca6429" (UID: "87b44f6d-d754-45da-8359-599eefca6429"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:02.206630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.206589 5855 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-bundle\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:02.206630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.206623 5855 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87b44f6d-d754-45da-8359-599eefca6429-util\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:02.206630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.206635 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jp8\" (UniqueName: \"kubernetes.io/projected/87b44f6d-d754-45da-8359-599eefca6429-kube-api-access-22jp8\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:02.853204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.853176 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" Oct 30 20:44:02.853594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.853176 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644z9vbg" event={"ID":"87b44f6d-d754-45da-8359-599eefca6429","Type":"ContainerDied","Data":"743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2"} Oct 30 20:44:02.853594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:02.853288 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="743c3c68091a93a5d6496a599781a2f44ee1ccf21a05fb293b73550bb01faae2" Oct 30 20:44:16.905429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905397 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh"] Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905838 5855 cpu_manager.go:414] "containerMap: removing container" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="pull" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905857 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="pull" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905869 5855 cpu_manager.go:414] "containerMap: removing container" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="extract" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905876 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="extract" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905889 5855 cpu_manager.go:414] "containerMap: removing container" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="util" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905898 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="util" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905915 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="extract" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905921 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="extract" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905948 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="pull" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905954 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="pull" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905967 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="util" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.905974 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="util" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.906053 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2337ed82-622d-42ea-982b-6b455f3ea15b" containerName="extract" Oct 30 20:44:16.906083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.906069 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b44f6d-d754-45da-8359-599eefca6429" containerName="extract" Oct 30 20:44:16.910184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.910158 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:16.953751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.953694 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh"] Oct 30 20:44:16.982682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.982616 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:44:16.982880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.982770 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:44:16.982880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.982820 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hv2f2\"" Oct 30 20:44:16.982880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.982826 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 20:44:16.983833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.983818 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:44:16.983881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:16.983858 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:44:17.031351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031316 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.031536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031374 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.031536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031451 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrttc\" (UniqueName: \"kubernetes.io/projected/dceda596-a161-410d-a9d8-bc672b975b22-kube-api-access-vrttc\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.031536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031517 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.031643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031553 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.031643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.031585 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.132974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.132930 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrttc\" (UniqueName: \"kubernetes.io/projected/dceda596-a161-410d-a9d8-bc672b975b22-kube-api-access-vrttc\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.132974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.132973 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.133224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.132992 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.133224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.133022 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.133224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.133059 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.133224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.133107 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.133476 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.133446 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.135585 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.135557 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dceda596-a161-410d-a9d8-bc672b975b22-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.135916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.135894 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.135995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.135900 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.136037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.135994 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dceda596-a161-410d-a9d8-bc672b975b22-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.145949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.145916 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrttc\" (UniqueName: \"kubernetes.io/projected/dceda596-a161-410d-a9d8-bc672b975b22-kube-api-access-vrttc\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh\" (UID: \"dceda596-a161-410d-a9d8-bc672b975b22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.223139 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.223049 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:17.521167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.521118 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh"] Oct 30 20:44:17.525490 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:44:17.525449 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceda596_a161_410d_a9d8_bc672b975b22.slice/crio-43cbed05555212ae12e30182f681afa713b437232f07336d6d5d10206980c31b WatchSource:0}: Error finding container 43cbed05555212ae12e30182f681afa713b437232f07336d6d5d10206980c31b: Status 404 returned error can't find the container with id 43cbed05555212ae12e30182f681afa713b437232f07336d6d5d10206980c31b Oct 30 20:44:17.911089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:17.911050 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" event={"ID":"dceda596-a161-410d-a9d8-bc672b975b22","Type":"ContainerStarted","Data":"43cbed05555212ae12e30182f681afa713b437232f07336d6d5d10206980c31b"} Oct 30 20:44:20.926960 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:20.926923 5855 generic.go:356] "Generic (PLEG): container finished" podID="dceda596-a161-410d-a9d8-bc672b975b22" containerID="e303dce265217432d245b78607d7cd443426065d302829a45b75068d61807b87" exitCode=0 Oct 30 20:44:20.927334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:20.926986 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" event={"ID":"dceda596-a161-410d-a9d8-bc672b975b22","Type":"ContainerDied","Data":"e303dce265217432d245b78607d7cd443426065d302829a45b75068d61807b87"} Oct 30 20:44:21.934340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:21.934309 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" event={"ID":"dceda596-a161-410d-a9d8-bc672b975b22","Type":"ContainerStarted","Data":"4b2ba16cf12a4434bbad9fe302f565e032e017e8ea917641e8b46143c681d3d6"} Oct 30 20:44:21.934793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:21.934382 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:21.995247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:21.995177 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" podStartSLOduration=2.088044636 podStartE2EDuration="5.995159046s" totalImagesPullingTime="3.346396598s" totalInitContainerRuntime="560.717812ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:21.994891102 +0000 UTC m=+275.353603846" watchObservedRunningTime="2026-10-30 20:44:21.995159046 +0000 UTC m=+275.353871794" Oct 30 20:44:30.924141 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:44:30.924062 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:44:32.939999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:32.939970 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-vdwbh" Oct 30 20:44:54.777155 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.777126 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp"] Oct 30 20:44:54.781320 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.781299 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.796432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.796402 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp"] Oct 30 20:44:54.844929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.844887 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f388c115-91a5-4c47-91fb-cb53bf16fe59-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.845101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.844947 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2z78\" (UniqueName: \"kubernetes.io/projected/f388c115-91a5-4c47-91fb-cb53bf16fe59-kube-api-access-x2z78\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.845101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.845033 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f388c115-91a5-4c47-91fb-cb53bf16fe59-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.875051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.874974 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 20:44:54.875051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.875036 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7wl7n\"" Oct 30 20:44:54.875521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.875491 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:44:54.945890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.945855 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f388c115-91a5-4c47-91fb-cb53bf16fe59-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.946121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.945918 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2z78\" (UniqueName: \"kubernetes.io/projected/f388c115-91a5-4c47-91fb-cb53bf16fe59-kube-api-access-x2z78\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.946121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.945952 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f388c115-91a5-4c47-91fb-cb53bf16fe59-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.948753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.948726 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f388c115-91a5-4c47-91fb-cb53bf16fe59-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.949726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.949704 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f388c115-91a5-4c47-91fb-cb53bf16fe59-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:54.969548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:54.969521 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2z78\" (UniqueName: \"kubernetes.io/projected/f388c115-91a5-4c47-91fb-cb53bf16fe59-kube-api-access-x2z78\") pod \"kube-auth-proxy-6c6fcc99c4-lnchp\" (UID: \"f388c115-91a5-4c47-91fb-cb53bf16fe59\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:55.095757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:55.095718 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" Oct 30 20:44:55.389837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:55.389808 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp"] Oct 30 20:44:55.392710 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:44:55.392677 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf388c115_91a5_4c47_91fb_cb53bf16fe59.slice/crio-818f67d2ab706e421589b48de6c240505a34ff5155567cad2d8c92fa4e171d61 WatchSource:0}: Error finding container 818f67d2ab706e421589b48de6c240505a34ff5155567cad2d8c92fa4e171d61: Status 404 returned error can't find the container with id 818f67d2ab706e421589b48de6c240505a34ff5155567cad2d8c92fa4e171d61 Oct 30 20:44:55.394586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:55.394568 5855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:44:56.082914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:56.082874 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" event={"ID":"f388c115-91a5-4c47-91fb-cb53bf16fe59","Type":"ContainerStarted","Data":"818f67d2ab706e421589b48de6c240505a34ff5155567cad2d8c92fa4e171d61"} Oct 30 20:44:58.094449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:44:58.094400 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" event={"ID":"f388c115-91a5-4c47-91fb-cb53bf16fe59","Type":"ContainerStarted","Data":"19479b7b24940f0794157472e6630c49360731fc7d8747af5a5583685363c2e5"} Oct 30 20:45:00.923307 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:45:00.923255 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:45:04.625627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.625559 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-lnchp" podStartSLOduration=8.09684626 podStartE2EDuration="10.625543119s" totalImagesPullingTime="2.528696859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:58.115641104 +0000 UTC m=+311.474353850" watchObservedRunningTime="2026-10-30 20:45:04.625543119 +0000 UTC m=+317.984255843" Oct 30 20:45:04.626052 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.626000 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 20:45:04.629438 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.629422 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.636075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.635978 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:45:04.636075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.636052 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:45:04.637350 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.637329 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7xcmg\"" Oct 30 20:45:04.643779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.643753 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 20:45:04.744874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.744838 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.745055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.744887 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jtn\" (UniqueName: \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.845670 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.845613 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.845860 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.845684 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jtn\" (UniqueName: \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.846076 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.846051 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.858416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.858382 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jtn\" (UniqueName: \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\") pod \"kuadrant-operator-controller-manager-678d87766f-dnn7r\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:04.942624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:04.942534 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:05.222801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:05.222777 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 20:45:05.225545 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:45:05.225515 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3aee86_ff23_4995_b442_2302bb1ee6ba.slice/crio-df37d2662e2a0e07ba8952d300b6a27f06b5999c8178a14fd83ffd465405f420 WatchSource:0}: Error finding container df37d2662e2a0e07ba8952d300b6a27f06b5999c8178a14fd83ffd465405f420: Status 404 returned error can't find the container with id df37d2662e2a0e07ba8952d300b6a27f06b5999c8178a14fd83ffd465405f420 Oct 30 20:45:06.127143 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:06.127111 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" event={"ID":"0c3aee86-ff23-4995-b442-2302bb1ee6ba","Type":"ContainerStarted","Data":"df37d2662e2a0e07ba8952d300b6a27f06b5999c8178a14fd83ffd465405f420"} Oct 30 20:45:10.149720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:10.149680 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" event={"ID":"0c3aee86-ff23-4995-b442-2302bb1ee6ba","Type":"ContainerStarted","Data":"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535"} Oct 30 20:45:10.150189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:10.149819 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:21.156930 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:21.156894 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 20:45:21.182932 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:21.182882 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" podStartSLOduration=12.728494211 podStartE2EDuration="17.182869224s" totalImagesPullingTime="4.454375013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:10.184637004 +0000 UTC m=+323.543349750" watchObservedRunningTime="2026-10-30 20:45:21.182869224 +0000 UTC m=+334.541581970" Oct 30 20:45:30.924472 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:45:30.924433 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:45:39.592896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.592862 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:45:39.598964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.598938 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.603795 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.603769 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-thfd4\"" Oct 30 20:45:39.606529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.606505 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:45:39.612952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.612932 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 20:45:39.618159 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.618127 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:45:39.659713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.659637 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-config-file\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.659914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.659858 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlm9d\" (UniqueName: \"kubernetes.io/projected/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-kube-api-access-mlm9d\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.761281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.761240 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-config-file\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.761468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.761348 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlm9d\" (UniqueName: \"kubernetes.io/projected/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-kube-api-access-mlm9d\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.761948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.761927 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-config-file\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.774288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.774264 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlm9d\" (UniqueName: \"kubernetes.io/projected/c28881ee-b9e8-42fc-9ff5-da0fffce60ec-kube-api-access-mlm9d\") pod \"limitador-limitador-7c9b87d574-tccqk\" (UID: \"c28881ee-b9e8-42fc-9ff5-da0fffce60ec\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:39.911729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:39.911616 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:40.184584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:40.184505 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:45:40.187159 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:45:40.187126 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28881ee_b9e8_42fc_9ff5_da0fffce60ec.slice/crio-4b03c0f0ae073d59024bb57fbd7f03f17b8fd9305907ebb08f0325be3d42b4fa WatchSource:0}: Error finding container 4b03c0f0ae073d59024bb57fbd7f03f17b8fd9305907ebb08f0325be3d42b4fa: Status 404 returned error can't find the container with id 4b03c0f0ae073d59024bb57fbd7f03f17b8fd9305907ebb08f0325be3d42b4fa Oct 30 20:45:40.275750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:40.275715 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" event={"ID":"c28881ee-b9e8-42fc-9ff5-da0fffce60ec","Type":"ContainerStarted","Data":"4b03c0f0ae073d59024bb57fbd7f03f17b8fd9305907ebb08f0325be3d42b4fa"} Oct 30 20:45:42.286060 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:42.286024 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" event={"ID":"c28881ee-b9e8-42fc-9ff5-da0fffce60ec","Type":"ContainerStarted","Data":"8b32ef6b5a54c0e01992e74e460a10b9e4d90f96442d46bcf210516b63a6126b"} Oct 30 20:45:42.286487 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:42.286148 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:42.309475 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:42.309414 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" podStartSLOduration=2.221584943 podStartE2EDuration="3.309395435s" totalImagesPullingTime="1.087810492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:42.308356775 +0000 UTC m=+355.667069521" watchObservedRunningTime="2026-10-30 20:45:42.309395435 +0000 UTC m=+355.668108192" Oct 30 20:45:51.499251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.499218 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:45:51.501832 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.501808 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.509578 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.509159 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 20:45:51.516288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.516230 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 20:45:51.518053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.518022 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 20:45:51.518566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.518541 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:45:51.518806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.518791 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pz7c6\"" Oct 30 20:45:51.679274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.679233 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg2j\" (UniqueName: \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.679486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.679306 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.780909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.780807 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg2j\" (UniqueName: \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.780909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.780860 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.783529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.783491 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:51.823219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:51.823185 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg2j\" (UniqueName: \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\") pod \"llmisvc-controller-manager-869d5dcc5d-4q2hk\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:52.026158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.026120 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm"] Oct 30 20:45:52.028408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.028381 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.045196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.045165 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm"] Oct 30 20:45:52.114227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.114189 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184205 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184249 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184273 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184292 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q99\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-kube-api-access-62q99\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184314 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184337 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.185692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.184360 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285435 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285478 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q99\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-kube-api-access-62q99\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285497 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285515 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285684 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285786 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.285919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.285828 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.286401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.286370 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.288210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.288184 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.288578 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.288554 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.288744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.288725 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.288853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.288832 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.298549 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.298521 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.298829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.298812 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q99\" (UniqueName: \"kubernetes.io/projected/7f4ed90c-75e2-457c-9d45-53a1a27b25fb-kube-api-access-62q99\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xvsjm\" (UID: \"7f4ed90c-75e2-457c-9d45-53a1a27b25fb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.339123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.339091 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:52.399645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.399609 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:45:52.623041 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.623019 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm"] Oct 30 20:45:52.625352 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:45:52.625317 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4ed90c_75e2_457c_9d45_53a1a27b25fb.slice/crio-f862343584f1a027321e29f3230355c387205e79cfd0f2b1d4ed4e7ce5ae2b8f WatchSource:0}: Error finding container f862343584f1a027321e29f3230355c387205e79cfd0f2b1d4ed4e7ce5ae2b8f: Status 404 returned error can't find the container with id f862343584f1a027321e29f3230355c387205e79cfd0f2b1d4ed4e7ce5ae2b8f Oct 30 20:45:52.627536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.627490 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:45:52.627637 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:52.627572 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:45:53.291283 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.291241 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-tccqk" Oct 30 20:45:53.340236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.340190 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" event={"ID":"7f4ed90c-75e2-457c-9d45-53a1a27b25fb","Type":"ContainerStarted","Data":"e0397a5b27d2de875f76ef0b7989f2563c219cf3c9a0061f10e94492c84d1dfb"} Oct 30 20:45:53.340434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.340265 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:53.340434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.340292 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" event={"ID":"7f4ed90c-75e2-457c-9d45-53a1a27b25fb","Type":"ContainerStarted","Data":"f862343584f1a027321e29f3230355c387205e79cfd0f2b1d4ed4e7ce5ae2b8f"} Oct 30 20:45:53.341644 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.341615 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" event={"ID":"cc852a38-7bc9-4c15-a4b3-3554735e9d11","Type":"ContainerStarted","Data":"fb5111bd760129a1bf38019c8414b77f2607b64bd2c30bd048110a3c1064d656"} Oct 30 20:45:53.342344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.342274 5855 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xvsjm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:45:53.342344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.342335 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" podUID="7f4ed90c-75e2-457c-9d45-53a1a27b25fb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:45:53.369780 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:53.369724 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" podStartSLOduration=1.369709692 podStartE2EDuration="1.369709692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:53.369290256 +0000 UTC m=+366.728003008" watchObservedRunningTime="2026-10-30 20:45:53.369709692 +0000 UTC m=+366.728422435" Oct 30 20:45:54.347281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.347253 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xvsjm" Oct 30 20:45:54.444284 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.444256 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:45:54.444465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.444446 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" containerID="cri-o://b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7" gracePeriod=30 Oct 30 20:45:54.683589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.683564 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:45:54.813138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813105 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813168 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813192 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813219 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813251 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813277 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813311 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\") pod \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\" (UID: \"b4fd6b93-5ccc-45ed-8ddd-e79e765048da\") " Oct 30 20:45:54.813772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.813749 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:54.816283 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816253 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:54.816423 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816377 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "kube-api-access-bdnnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:54.816485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816427 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:54.816485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816456 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:54.816738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816714 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:45:54.816809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.816778 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts" pod "b4fd6b93-5ccc-45ed-8ddd-e79e765048da" (UID: "b4fd6b93-5ccc-45ed-8ddd-e79e765048da"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915275 5855 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-kubeconfig\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915361 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915380 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdnnd\" (UniqueName: \"kubernetes.io/projected/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-kube-api-access-bdnnd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915399 5855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-ca-configmap\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915414 5855 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-cacerts\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915429 5855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-istio-csr-dns-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:54.915452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:54.915445 5855 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fd6b93-5ccc-45ed-8ddd-e79e765048da-local-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:55.352343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.352307 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" event={"ID":"cc852a38-7bc9-4c15-a4b3-3554735e9d11","Type":"ContainerStarted","Data":"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98"} Oct 30 20:45:55.352856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.352433 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:45:55.353788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.353765 5855 generic.go:356] "Generic (PLEG): container finished" podID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerID="b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7" exitCode=0 Oct 30 20:45:55.353896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.353804 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" Oct 30 20:45:55.353896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.353851 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" event={"ID":"b4fd6b93-5ccc-45ed-8ddd-e79e765048da","Type":"ContainerDied","Data":"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7"} Oct 30 20:45:55.353976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.353898 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft" event={"ID":"b4fd6b93-5ccc-45ed-8ddd-e79e765048da","Type":"ContainerDied","Data":"4b0178a6294745c78b5cdba7f7001ddf33420470bc3fd8fdc3d2702b1564b0f8"} Oct 30 20:45:55.353976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.353923 5855 scope.go:122] "RemoveContainer" containerID="b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7" Oct 30 20:45:55.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.369498 5855 scope.go:122] "RemoveContainer" containerID="b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7" Oct 30 20:45:55.369858 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:45:55.369835 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7\": container with ID starting with b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7 not found: ID does not exist" containerID="b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7" Oct 30 20:45:55.369939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.369868 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7"} err="failed to get container status \"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7\": rpc error: code = NotFound desc = could not find container \"b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7\": container with ID starting with b2df4eebffdcf036062a2002db5108f045654ffad3f9ed203a8c39cfea35c4e7 not found: ID does not exist" Oct 30 20:45:55.378541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.378493 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" podStartSLOduration=2.395348981 podStartE2EDuration="4.378480342s" totalImagesPullingTime="1.983131361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:55.377326761 +0000 UTC m=+368.736039508" watchObservedRunningTime="2026-10-30 20:45:55.378480342 +0000 UTC m=+368.737193130" Oct 30 20:45:55.392310 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.392260 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:45:55.401403 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:55.401349 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kpwft"] Oct 30 20:45:56.779004 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:45:56.778969 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" path="/var/lib/kubelet/pods/b4fd6b93-5ccc-45ed-8ddd-e79e765048da/volumes" Oct 30 20:46:00.922199 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:46:00.922153 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:46:26.364588 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:46:26.364549 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:46:30.924569 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:46:30.924529 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:47:00.925319 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:47:00.925277 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:47:05.845282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:05.845238 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:47:18.552349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.552313 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb"] Oct 30 20:47:18.552854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.552675 5855 cpu_manager.go:414] "containerMap: removing container" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" Oct 30 20:47:18.552854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.552689 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" Oct 30 20:47:18.552854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.552754 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fd6b93-5ccc-45ed-8ddd-e79e765048da" containerName="discovery" Oct 30 20:47:18.556058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.556039 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.564156 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.564130 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb"] Oct 30 20:47:18.651619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.651573 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pxv\" (UniqueName: \"kubernetes.io/projected/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-kube-api-access-77pxv\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.651843 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.651742 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.753011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.752973 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.753194 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.753040 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pxv\" (UniqueName: \"kubernetes.io/projected/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-kube-api-access-77pxv\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.755819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.755788 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.766687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.766645 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pxv\" (UniqueName: \"kubernetes.io/projected/6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30-kube-api-access-77pxv\") pod \"llmisvc-controller-manager-6466c554d8-sdjpb\" (UID: \"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:18.869957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:18.869864 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:19.142882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:19.142810 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb"] Oct 30 20:47:19.145950 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:47:19.145923 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8a22c7_ddeb_4e31_8cab_54d6e40a9c30.slice/crio-c6edb36c4a480f6e8682c5dd7d449a4a45b073cd4402804286bf443f2b68338f WatchSource:0}: Error finding container c6edb36c4a480f6e8682c5dd7d449a4a45b073cd4402804286bf443f2b68338f: Status 404 returned error can't find the container with id c6edb36c4a480f6e8682c5dd7d449a4a45b073cd4402804286bf443f2b68338f Oct 30 20:47:19.729927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:19.729890 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" event={"ID":"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30","Type":"ContainerStarted","Data":"7fe147817f97572d8d577fd1650bd205f492d63acf6043d04edb02f7581ab439"} Oct 30 20:47:19.730345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:19.729975 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" event={"ID":"6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30","Type":"ContainerStarted","Data":"c6edb36c4a480f6e8682c5dd7d449a4a45b073cd4402804286bf443f2b68338f"} Oct 30 20:47:19.730345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:19.730090 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:19.751474 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:19.751371 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" podStartSLOduration=1.415502101 podStartE2EDuration="1.75135949s" totalImagesPullingTime="335.857389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:19.751190139 +0000 UTC m=+453.109902886" watchObservedRunningTime="2026-10-30 20:47:19.75135949 +0000 UTC m=+453.110072235" Oct 30 20:47:30.923881 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:47:30.923792 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:47:50.737359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:50.737326 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdjpb" Oct 30 20:47:50.785711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:50.785678 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:47:50.785962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:50.785936 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" podUID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" containerName="manager" containerID="cri-o://4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98" gracePeriod=30 Oct 30 20:47:51.036743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.036715 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:47:51.132110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.132075 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\" (UniqueName: \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\") pod \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " Oct 30 20:47:51.132290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.132145 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\" (UniqueName: \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\") pod \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\" (UID: \"cc852a38-7bc9-4c15-a4b3-3554735e9d11\") " Oct 30 20:47:51.134367 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.134330 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert" pod "cc852a38-7bc9-4c15-a4b3-3554735e9d11" (UID: "cc852a38-7bc9-4c15-a4b3-3554735e9d11"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:51.134471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.134330 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j" pod "cc852a38-7bc9-4c15-a4b3-3554735e9d11" (UID: "cc852a38-7bc9-4c15-a4b3-3554735e9d11"). InnerVolumeSpecName "kube-api-access-smg2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:51.233733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.233699 5855 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc852a38-7bc9-4c15-a4b3-3554735e9d11-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:51.233733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.233729 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg2j\" (UniqueName: \"kubernetes.io/projected/cc852a38-7bc9-4c15-a4b3-3554735e9d11-kube-api-access-smg2j\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:51.874460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.874422 5855 generic.go:356] "Generic (PLEG): container finished" podID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" containerID="4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98" exitCode=0 Oct 30 20:47:51.874938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.874487 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" Oct 30 20:47:51.874938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.874509 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" event={"ID":"cc852a38-7bc9-4c15-a4b3-3554735e9d11","Type":"ContainerDied","Data":"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98"} Oct 30 20:47:51.874938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.874576 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk" event={"ID":"cc852a38-7bc9-4c15-a4b3-3554735e9d11","Type":"ContainerDied","Data":"fb5111bd760129a1bf38019c8414b77f2607b64bd2c30bd048110a3c1064d656"} Oct 30 20:47:51.874938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.874598 5855 scope.go:122] "RemoveContainer" containerID="4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98" Oct 30 20:47:51.884057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.884038 5855 scope.go:122] "RemoveContainer" containerID="4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98" Oct 30 20:47:51.884412 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:47:51.884393 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98\": container with ID starting with 4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98 not found: ID does not exist" containerID="4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98" Oct 30 20:47:51.884480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.884425 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98"} err="failed to get container status \"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98\": rpc error: code = NotFound desc = could not find container \"4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98\": container with ID starting with 4557fb29dfe50c07d27efbf61483f84f011c78103aeb0c8ade72dd7372c5ec98 not found: ID does not exist" Oct 30 20:47:51.899810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.899773 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:47:51.904062 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:51.904035 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4q2hk"] Oct 30 20:47:52.777799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:47:52.777768 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" path="/var/lib/kubelet/pods/cc852a38-7bc9-4c15-a4b3-3554735e9d11/volumes" Oct 30 20:48:00.925077 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:48:00.925035 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:48:10.714702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:10.714645 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:48:13.458083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:13.458045 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:48:16.468158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:16.468124 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:48:21.777563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:21.777528 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:48:30.925029 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:48:30.924990 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:48:35.243766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.243731 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 20:48:35.244306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.244230 5855 cpu_manager.go:414] "containerMap: removing container" podUID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" containerName="manager" Oct 30 20:48:35.244306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.244246 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" containerName="manager" Oct 30 20:48:35.244414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.244400 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc852a38-7bc9-4c15-a4b3-3554735e9d11" containerName="manager" Oct 30 20:48:35.247785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.247768 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.259331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.259303 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 20:48:35.330240 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.330207 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vgmtz\"" Oct 30 20:48:35.344841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.344803 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.344841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.344852 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.344969 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345030 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345116 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345146 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345186 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9vs\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345254 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.345340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.345290 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446110 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446154 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446195 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446217 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446252 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446286 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446324 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446347 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446385 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9vs\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446824 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446617 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446824 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446693 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.446956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.446935 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.447037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.447018 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.447139 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.447117 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.448815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.448789 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.449203 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.449181 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.460009 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.459971 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.460164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.460128 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9vs\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-ph82z\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.562225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.562191 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:35.892449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:35.892421 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 20:48:35.895067 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:48:35.895024 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b19e8fb_247e_4f4e_850b_60074ef710ab.slice/crio-9899e15dc01c7740955c27df7e2ea477393c1b92b526be6d3ec6990b2f2c2754 WatchSource:0}: Error finding container 9899e15dc01c7740955c27df7e2ea477393c1b92b526be6d3ec6990b2f2c2754: Status 404 returned error can't find the container with id 9899e15dc01c7740955c27df7e2ea477393c1b92b526be6d3ec6990b2f2c2754 Oct 30 20:48:36.074972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:36.074939 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" event={"ID":"6b19e8fb-247e-4f4e-850b-60074ef710ab","Type":"ContainerStarted","Data":"9899e15dc01c7740955c27df7e2ea477393c1b92b526be6d3ec6990b2f2c2754"} Oct 30 20:48:39.313724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:39.313688 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:39.314157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:39.313773 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:39.314157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:39.313820 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:40.106258 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:40.106140 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" event={"ID":"6b19e8fb-247e-4f4e-850b-60074ef710ab","Type":"ContainerStarted","Data":"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f"} Oct 30 20:48:40.563354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:40.563316 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:40.565074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:40.565048 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:40.565193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:40.565112 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:41.563350 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:41.563307 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:41.563527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:41.563369 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:42.563445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:42.563400 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:42.563623 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:42.563466 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:43.563171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:43.563134 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:43.565408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:43.563187 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:44.563410 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:44.563373 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:44.563598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:44.563426 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:45.562723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:45.562680 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:45.563128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:45.562755 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:46.562905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:46.562847 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Oct 30 20:48:46.563464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:46.562944 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Oct 30 20:48:48.564116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:48.564068 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:48.564611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:48.564145 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:49.604886 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.604838 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:49.605335 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.604912 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:49.759744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.759712 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:49.760074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.760031 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:49.760827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.760799 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 20:48:49.789036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:49.788983 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podStartSLOduration=11.373040671 podStartE2EDuration="14.788970549s" totalImagesPullingTime="3.415929878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:40.200287166 +0000 UTC m=+533.558999918" watchObservedRunningTime="2026-10-30 20:48:49.788970549 +0000 UTC m=+543.147683306" Oct 30 20:48:50.241043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.241001 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 20:48:50.247158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.247128 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.266588 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.266556 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 20:48:50.301709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.301646 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.301922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.301773 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.301922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.301813 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.301922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.301861 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.302066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.301974 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.302066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.302013 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.302066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.302040 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.302161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.302109 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zgd\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.302161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.302148 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.402931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.402888 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.402973 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403005 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403024 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403050 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403076 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403108 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403428 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403143 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403428 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403179 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zgd\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403565 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403642 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403704 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403763 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.403826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.403770 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.405488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.405466 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.405890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.405873 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.433196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.433158 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zgd\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.435442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.435414 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-spcjv\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.561035 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.560999 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:50.844578 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.844550 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 20:48:50.845872 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:48:50.845847 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180b2492_c057_4886_89f9_d748be169507.slice/crio-0ced0ff21126487597361b7f5337f1c597debb1de9fd341de2d9853a7253734d WatchSource:0}: Error finding container 0ced0ff21126487597361b7f5337f1c597debb1de9fd341de2d9853a7253734d: Status 404 returned error can't find the container with id 0ced0ff21126487597361b7f5337f1c597debb1de9fd341de2d9853a7253734d Oct 30 20:48:50.848122 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.848092 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:50.848183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.848163 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:50.848218 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:50.848193 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:48:51.158126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.158026 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" event={"ID":"180b2492-c057-4886-89f9-d748be169507","Type":"ContainerStarted","Data":"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21"} Oct 30 20:48:51.158126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.158084 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" event={"ID":"180b2492-c057-4886-89f9-d748be169507","Type":"ContainerStarted","Data":"0ced0ff21126487597361b7f5337f1c597debb1de9fd341de2d9853a7253734d"} Oct 30 20:48:51.194682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.194609 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podStartSLOduration=1.194594363 podStartE2EDuration="1.194594363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:51.193892064 +0000 UTC m=+544.552604809" watchObservedRunningTime="2026-10-30 20:48:51.194594363 +0000 UTC m=+544.553307106" Oct 30 20:48:51.562121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.562092 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:51.563605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.563579 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:51.563700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:51.563632 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:52.562421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:52.562385 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:52.562885 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:52.562450 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:53.561741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:53.561706 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:53.561963 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:53.561783 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:54.561927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:54.561846 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:54.561927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:54.561894 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:55.562194 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:55.562155 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:55.562586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:55.562212 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:56.561996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:56.561954 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:56.562182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:56.562015 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:57.562106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:57.562067 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 20:48:57.562542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:57.562128 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 20:48:59.097142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:59.097108 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:59.097624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:59.097540 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:48:59.098242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:48:59.098225 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 20:49:00.924033 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:49:00.923999 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:49:30.924424 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:49:30.924387 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:49:46.831085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.831026 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:49:46.839200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.839167 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.844823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.844792 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cb60a2-openshift-default-dockercfg-zl72q\"" Oct 30 20:49:46.861671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.861592 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:49:46.915000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.914963 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915007 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915029 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915148 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915186 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915207 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915294 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915313 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:46.915507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:46.915329 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8q2\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016285 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016327 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016353 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016393 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016432 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016457 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016486 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016508 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.016575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016531 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8q2\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.017047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016810 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.017047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.016909 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.017161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.017056 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.017161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.017065 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.017258 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.017198 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.019592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.019563 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.019782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.019761 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.034266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.034231 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.034266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.034260 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8q2\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.167093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.166992 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:47.484161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.484101 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:49:47.489902 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.489857 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:49:47.490040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.489957 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:49:47.490040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:47.489998 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:49:48.429865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:48.429821 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerStarted","Data":"d610a9b5a93f4e5bc94d2dee0ed0537fe8a2151a6e3d624275237d999431cd3b"} Oct 30 20:49:48.430390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:48.429882 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerStarted","Data":"e47cb3db973c112b306a9c7af41865ff1edde0fa5d049a26e2d7ac43589fa6f4"} Oct 30 20:49:48.457173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:48.457116 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podStartSLOduration=2.457101054 podStartE2EDuration="2.457101054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:48.454976045 +0000 UTC m=+601.813688816" watchObservedRunningTime="2026-10-30 20:49:48.457101054 +0000 UTC m=+601.815813800" Oct 30 20:49:49.167702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:49.167627 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:49.173870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:49.173836 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:49.435129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:49.435043 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:49.436137 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:49.436114 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:49:53.491715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:53.491672 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:49:57.660086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:49:57.660049 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:00.936494 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:50:00.936444 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:50:01.167625 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.167591 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 20:50:01.171436 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.171407 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.184401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.184346 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 20:50:01.197908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.197814 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f2ed1e\"" Oct 30 20:50:01.254975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.254949 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f2ed1e-dockercfg-hwznm\"" Oct 30 20:50:01.265413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.265380 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.265593 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.265432 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbjm\" (UniqueName: \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.367047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.366984 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbjm\" (UniqueName: \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.367259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.367134 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.368287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.368251 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.418344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.418313 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbjm\" (UniqueName: \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\") pod \"payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.498730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.493143 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:01.823700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.823646 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 20:50:01.826538 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:01.826505 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ff0d13_613d_4e61_9309_717c5dd63fdc.slice/crio-2acad65ac71afa8860cf19d8e513236358ecb02240510c1c5eef2ebab4c0f957 WatchSource:0}: Error finding container 2acad65ac71afa8860cf19d8e513236358ecb02240510c1c5eef2ebab4c0f957: Status 404 returned error can't find the container with id 2acad65ac71afa8860cf19d8e513236358ecb02240510c1c5eef2ebab4c0f957 Oct 30 20:50:01.828624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:01.828606 5855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:50:02.104468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.104383 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:50:02.109267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.109225 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.127055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.126921 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"openshift-service-ca.crt\"" Oct 30 20:50:02.127055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.126998 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"default-dockercfg-rt952\"" Oct 30 20:50:02.127055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.126998 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"simulator-w0-ad7ff0-kserve-self-signed-certs\"" Oct 30 20:50:02.129707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.128458 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"kube-root-ca.crt\"" Oct 30 20:50:02.130525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.130502 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:50:02.174330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174279 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.174566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174341 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwplb\" (UniqueName: \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.174566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174378 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.174566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174436 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.174566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174467 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.174833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.174580 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275735 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275791 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275819 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwplb\" (UniqueName: \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275846 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275905 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.275975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.275929 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.276249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.276227 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.276309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.276290 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.276497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.276470 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.278597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.278564 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.279059 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.279037 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.335530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.335500 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwplb\" (UniqueName: \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\") pod \"simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.409001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.408918 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:02.442206 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.442157 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:02.506477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.506440 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" event={"ID":"d0ff0d13-613d-4e61-9309-717c5dd63fdc","Type":"ContainerStarted","Data":"2acad65ac71afa8860cf19d8e513236358ecb02240510c1c5eef2ebab4c0f957"} Oct 30 20:50:02.798693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:02.798603 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:50:02.805040 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:02.804994 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f2bef3_0671_4bec_bb3d_c5bacab55cde.slice/crio-14ae6509cdc72aad0ede5f4dd95f412f8b72c6af019681a84fe60982519bbd28 WatchSource:0}: Error finding container 14ae6509cdc72aad0ede5f4dd95f412f8b72c6af019681a84fe60982519bbd28: Status 404 returned error can't find the container with id 14ae6509cdc72aad0ede5f4dd95f412f8b72c6af019681a84fe60982519bbd28 Oct 30 20:50:03.516243 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:03.516198 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" event={"ID":"06f2bef3-0671-4bec-bb3d-c5bacab55cde","Type":"ContainerStarted","Data":"14ae6509cdc72aad0ede5f4dd95f412f8b72c6af019681a84fe60982519bbd28"} Oct 30 20:50:05.538314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:05.538256 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" event={"ID":"d0ff0d13-613d-4e61-9309-717c5dd63fdc","Type":"ContainerStarted","Data":"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082"} Oct 30 20:50:05.538835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:05.538339 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:05.539976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:05.539952 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" event={"ID":"06f2bef3-0671-4bec-bb3d-c5bacab55cde","Type":"ContainerStarted","Data":"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca"} Oct 30 20:50:05.588444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:05.588393 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" podStartSLOduration=1.6008241970000001 podStartE2EDuration="4.588378471s" totalImagesPullingTime="2.987554274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:05.586788003 +0000 UTC m=+618.945500748" watchObservedRunningTime="2026-10-30 20:50:05.588378471 +0000 UTC m=+618.947091234" Oct 30 20:50:05.636089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:05.636012 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podStartSLOduration=1.83305257 podStartE2EDuration="3.635996178s" totalImagesPullingTime="1.802943608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:05.635344182 +0000 UTC m=+618.994056929" watchObservedRunningTime="2026-10-30 20:50:05.635996178 +0000 UTC m=+618.994708925" Oct 30 20:50:06.443377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:06.443335 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:09.100444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:09.100405 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:11.781826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.781795 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:50:11.785841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.785815 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.797458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.797430 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"openshift-service-ca.crt\"" Oct 30 20:50:11.797726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.797473 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"kube-root-ca.crt\"" Oct 30 20:50:11.797726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.797593 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"simulator-w2-db534b-kserve-self-signed-certs\"" Oct 30 20:50:11.799107 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.799085 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"default-dockercfg-z8rc4\"" Oct 30 20:50:11.811403 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.811360 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:50:11.912422 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.912380 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.913056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.912982 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.913332 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.913313 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.913417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.913400 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.913532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.913508 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:11.913583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:11.913550 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlpg\" (UniqueName: \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014529 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014622 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014694 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014718 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014762 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.014999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.014798 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlpg\" (UniqueName: \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.015118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.015087 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.015464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.015431 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.015606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.015541 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.018909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.018881 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.019810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.019794 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.037471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.037401 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlpg\" (UniqueName: \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\") pod \"simulator-w2-db534b-kserve-6f5f788cb5-s5ftf\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.099087 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.099056 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:12.393467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.393434 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:50:12.396318 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:12.396291 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ddef63a_06bb_44b2_8503_0868fc054338.slice/crio-24af095dd57e3572d1d76183ba6eb13d1e527ef215b60bfb2ddf9d298dd7e4ed WatchSource:0}: Error finding container 24af095dd57e3572d1d76183ba6eb13d1e527ef215b60bfb2ddf9d298dd7e4ed: Status 404 returned error can't find the container with id 24af095dd57e3572d1d76183ba6eb13d1e527ef215b60bfb2ddf9d298dd7e4ed Oct 30 20:50:12.444022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.443138 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:12.459897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.459564 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:12.578176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.578131 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" event={"ID":"7ddef63a-06bb-44b2-8503-0868fc054338","Type":"ContainerStarted","Data":"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8"} Oct 30 20:50:12.578176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.578172 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" event={"ID":"7ddef63a-06bb-44b2-8503-0868fc054338","Type":"ContainerStarted","Data":"24af095dd57e3572d1d76183ba6eb13d1e527ef215b60bfb2ddf9d298dd7e4ed"} Oct 30 20:50:12.590743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.590714 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:50:12.602948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:12.602887 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podStartSLOduration=1.6028674 podStartE2EDuration="1.6028674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:12.599168154 +0000 UTC m=+625.957880902" watchObservedRunningTime="2026-10-30 20:50:12.6028674 +0000 UTC m=+625.961580150" Oct 30 20:50:13.099306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:13.099248 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:16.548922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:16.548887 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 20:50:18.239256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.239211 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:50:18.243827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.243800 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.256380 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.256351 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"kube-root-ca.crt\"" Oct 30 20:50:18.257631 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.257606 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"openshift-service-ca.crt\"" Oct 30 20:50:18.268854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.268813 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:50:18.288575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.288537 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"simulator-w5-14cca2-kserve-self-signed-certs\"" Oct 30 20:50:18.289624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.289024 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"default-dockercfg-lspws\"" Oct 30 20:50:18.291927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.289440 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.292207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.292188 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.292350 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.292332 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.292504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.292484 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.292630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.292614 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.292804 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.292784 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmgh\" (UniqueName: \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.398192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.398147 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.398984 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.398961 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.399507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.399476 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.400018 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.400001 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.400463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.400444 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmgh\" (UniqueName: \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.400968 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.400949 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.401155 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.400382 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.401255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.399942 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.401586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.401567 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.404992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.404973 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.411721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.411695 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.414742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.414714 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmgh\" (UniqueName: \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\") pod \"simulator-w5-14cca2-kserve-55749b4f64-mgdz5\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.557925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.557884 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:18.933321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:18.933271 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:50:18.936415 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:18.936377 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57cf0bfd_a20c_4aef_bf46_096da4c9dcff.slice/crio-f55ace2127e525653f26a43ba8927e6588ad061dfcf4d5483b187d8d81a23610 WatchSource:0}: Error finding container f55ace2127e525653f26a43ba8927e6588ad061dfcf4d5483b187d8d81a23610: Status 404 returned error can't find the container with id f55ace2127e525653f26a43ba8927e6588ad061dfcf4d5483b187d8d81a23610 Oct 30 20:50:19.653755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:19.653712 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" event={"ID":"57cf0bfd-a20c-4aef-bf46-096da4c9dcff","Type":"ContainerStarted","Data":"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0"} Oct 30 20:50:19.653755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:19.653763 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" event={"ID":"57cf0bfd-a20c-4aef-bf46-096da4c9dcff","Type":"ContainerStarted","Data":"f55ace2127e525653f26a43ba8927e6588ad061dfcf4d5483b187d8d81a23610"} Oct 30 20:50:19.693725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:19.693641 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podStartSLOduration=1.693619724 podStartE2EDuration="1.693619724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:19.692069485 +0000 UTC m=+633.050782259" watchObservedRunningTime="2026-10-30 20:50:19.693619724 +0000 UTC m=+633.052332471" Oct 30 20:50:20.558620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:20.558581 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:21.396687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.382749 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 20:50:21.424724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.424233 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.426162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.426133 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 20:50:21.436832 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.436798 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"default-dockercfg-qmhc7\"" Oct 30 20:50:21.438171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.438031 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"simulator-w1-b8aee1-kserve-self-signed-certs\"" Oct 30 20:50:21.438171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.438107 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"kube-root-ca.crt\"" Oct 30 20:50:21.438695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.438576 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"openshift-service-ca.crt\"" Oct 30 20:50:21.451344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.451298 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.451924 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.451900 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.452106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.452088 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.452277 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.452246 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26k69\" (UniqueName: \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.452468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.452447 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.452543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.452525 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.553686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553618 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.553686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553678 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.553926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553810 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.553926 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553908 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.554044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553938 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.554044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.553973 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26k69\" (UniqueName: \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.554158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.554076 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.554213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.554177 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.554266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.554230 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.556430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.556405 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.556871 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.556855 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.566939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.566913 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26k69\" (UniqueName: \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\") pod \"simulator-w1-b8aee1-kserve-67fc4dddcd-952mv\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:21.746172 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:21.746073 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:22.039222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.039193 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 20:50:22.042114 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:22.042073 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod690b2c59_3c3e_43b1_88b1_29aaff3a00d8.slice/crio-c87adf928fb23f1cf137816789fa4228567b19141d7fdf3e006cade8689e7321 WatchSource:0}: Error finding container c87adf928fb23f1cf137816789fa4228567b19141d7fdf3e006cade8689e7321: Status 404 returned error can't find the container with id c87adf928fb23f1cf137816789fa4228567b19141d7fdf3e006cade8689e7321 Oct 30 20:50:22.100358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.099850 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:22.124013 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.123980 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:22.616786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.616746 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:22.671771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.671723 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" event={"ID":"690b2c59-3c3e-43b1-88b1-29aaff3a00d8","Type":"ContainerStarted","Data":"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a"} Oct 30 20:50:22.671953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.671780 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" event={"ID":"690b2c59-3c3e-43b1-88b1-29aaff3a00d8","Type":"ContainerStarted","Data":"c87adf928fb23f1cf137816789fa4228567b19141d7fdf3e006cade8689e7321"} Oct 30 20:50:22.686194 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.686164 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:50:22.746427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.746397 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:22.789238 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:22.789165 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podStartSLOduration=1.789140368 podStartE2EDuration="1.789140368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:22.788024587 +0000 UTC m=+636.146737335" watchObservedRunningTime="2026-10-30 20:50:22.789140368 +0000 UTC m=+636.147853115" Oct 30 20:50:25.688368 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.688329 5855 generic.go:356] "Generic (PLEG): container finished" podID="9917567c-d501-4d54-9067-a259a98d9b44" containerID="d610a9b5a93f4e5bc94d2dee0ed0537fe8a2151a6e3d624275237d999431cd3b" exitCode=0 Oct 30 20:50:25.688952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.688377 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerDied","Data":"d610a9b5a93f4e5bc94d2dee0ed0537fe8a2151a6e3d624275237d999431cd3b"} Oct 30 20:50:25.688952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.688855 5855 scope.go:122] "RemoveContainer" containerID="d610a9b5a93f4e5bc94d2dee0ed0537fe8a2151a6e3d624275237d999431cd3b" Oct 30 20:50:25.690309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.690274 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:25.690398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.690364 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:25.690436 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:25.690408 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:50:26.707321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:26.707283 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerStarted","Data":"00dcdf6a5ba33848ce2a6cb86953b2aac01afe2494b0a4794fd0468bf5cb9eb5"} Oct 30 20:50:27.167938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:27.167909 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:50:27.168387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:27.168325 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 20:50:27.168488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:27.168441 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 20:50:28.167669 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:28.167609 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 20:50:28.168278 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:28.167720 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 20:50:28.558700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:28.558614 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:28.571607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:28.571576 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:28.738970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:28.738936 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:50:29.167789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:29.167759 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 20:50:29.168287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:29.168255 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 20:50:30.168026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:30.167987 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 20:50:30.168474 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:30.168066 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 20:50:30.936463 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:50:30.936425 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:50:31.169680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:31.167573 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 20:50:31.169680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:31.167627 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 20:50:31.746405 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:31.746368 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:31.759850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:31.759812 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:32.172527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:32.172496 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:50:32.173018 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:32.172788 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:50:32.173936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:32.173913 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:50:32.757765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:32.757735 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:50:33.429366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.429322 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:50:33.438109 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.438079 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.444105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.444077 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:50:33.448106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.448085 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"kube-root-ca.crt\"" Oct 30 20:50:33.455276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.455243 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"openshift-service-ca.crt\"" Oct 30 20:50:33.458826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.458796 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"simulator-w6-3d909d-kserve-self-signed-certs\"" Oct 30 20:50:33.458826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.458810 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"default-dockercfg-2wvvq\"" Oct 30 20:50:33.521243 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521203 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.521458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521270 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.521458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521375 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.521458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521438 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.521776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521550 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mts\" (UniqueName: \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.521776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.521604 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622681 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622752 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mts\" (UniqueName: \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622784 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622810 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622846 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.622947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.622902 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.623245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.623200 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.623302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.623255 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.623474 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.623434 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.625830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.625798 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.625984 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.625845 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.646808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.641923 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mts\" (UniqueName: \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\") pod \"simulator-w6-3d909d-kserve-b494fdb44-pd9cp\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:33.756739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:33.756633 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:34.045950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:34.045357 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:50:34.050019 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:34.049974 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5da1eae_02ea_476c_b086_247197542e8f.slice/crio-3dc2f6d2983428031e8663c5e5bf0bd7f3c5da89a20d49b613a980bf23caf124 WatchSource:0}: Error finding container 3dc2f6d2983428031e8663c5e5bf0bd7f3c5da89a20d49b613a980bf23caf124: Status 404 returned error can't find the container with id 3dc2f6d2983428031e8663c5e5bf0bd7f3c5da89a20d49b613a980bf23caf124 Oct 30 20:50:34.755758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:34.755708 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" event={"ID":"e5da1eae-02ea-476c-b086-247197542e8f","Type":"ContainerStarted","Data":"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb"} Oct 30 20:50:34.755758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:34.755759 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" event={"ID":"e5da1eae-02ea-476c-b086-247197542e8f","Type":"ContainerStarted","Data":"3dc2f6d2983428031e8663c5e5bf0bd7f3c5da89a20d49b613a980bf23caf124"} Oct 30 20:50:34.757096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:34.757071 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:34.782213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:34.782158 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podStartSLOduration=1.782140578 podStartE2EDuration="1.782140578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:34.781808479 +0000 UTC m=+648.140521224" watchObservedRunningTime="2026-10-30 20:50:34.782140578 +0000 UTC m=+648.140853324" Oct 30 20:50:37.161841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:37.161803 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:39.379229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.379193 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 20:50:39.387792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.387763 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.395502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.395445 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 20:50:39.399472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.399394 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"default-dockercfg-2nmmh\"" Oct 30 20:50:39.399570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.399490 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"kube-root-ca.crt\"" Oct 30 20:50:39.399624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.399568 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"openshift-service-ca.crt\"" Oct 30 20:50:39.399691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.399684 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"simulator-w4-f2ed1e-kserve-self-signed-certs\"" Oct 30 20:50:39.497270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497224 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5sdk\" (UniqueName: \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.497475 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497294 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.497534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497501 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.497575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497536 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.497620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497574 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.497798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.497773 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598470 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598533 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5sdk\" (UniqueName: \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598579 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598647 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598689 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.598760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.598723 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.599300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.599274 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.599437 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.599399 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.599561 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.599542 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.601513 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.601484 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.608428 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.608397 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.613019 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.612992 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5sdk\" (UniqueName: \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\") pod \"simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.705465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.705372 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:39.988821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:39.988799 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 20:50:39.990901 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:39.990873 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod916e593c_f5ee_47a9_bcc7_7167382c4d4b.slice/crio-68386f544bf815e1772cf5be236a2738bfdaa5ccca4198895c33af44a311f889 WatchSource:0}: Error finding container 68386f544bf815e1772cf5be236a2738bfdaa5ccca4198895c33af44a311f889: Status 404 returned error can't find the container with id 68386f544bf815e1772cf5be236a2738bfdaa5ccca4198895c33af44a311f889 Oct 30 20:50:40.274158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.274079 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:50:40.278352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.278321 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.294359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.293627 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:50:40.294359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.294220 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"premium-w2-db534b-kserve-self-signed-certs\"" Oct 30 20:50:40.408204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.408155 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.408824 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.408794 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.409340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.409010 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.409340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.409048 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.409340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.409099 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95gj\" (UniqueName: \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.409340 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.409196 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510364 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510405 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510440 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95gj\" (UniqueName: \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510501 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510553 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510592 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.510941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.510919 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.511135 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.511109 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.511223 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.511181 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.513292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.513265 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.513549 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.513531 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.523397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.523366 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95gj\" (UniqueName: \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\") pod \"premium-w2-db534b-kserve-5dc67b9fb8-dgfqn\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.608091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.607124 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:40.794283 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.794244 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" event={"ID":"916e593c-f5ee-47a9-bcc7-7167382c4d4b","Type":"ContainerStarted","Data":"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2"} Oct 30 20:50:40.794486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.794297 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" event={"ID":"916e593c-f5ee-47a9-bcc7-7167382c4d4b","Type":"ContainerStarted","Data":"68386f544bf815e1772cf5be236a2738bfdaa5ccca4198895c33af44a311f889"} Oct 30 20:50:40.814183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.814122 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podStartSLOduration=1.814106497 podStartE2EDuration="1.814106497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:40.813380005 +0000 UTC m=+654.172092756" watchObservedRunningTime="2026-10-30 20:50:40.814106497 +0000 UTC m=+654.172819302" Oct 30 20:50:40.896075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:40.896050 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:50:40.897993 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:40.897959 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a3b9107_6606_4a8c_806f_da7e770d09ae.slice/crio-36fb9e9b21ada0ba52e1b1fcce7e038446828d811aeb4202e5519cb5245e9962 WatchSource:0}: Error finding container 36fb9e9b21ada0ba52e1b1fcce7e038446828d811aeb4202e5519cb5245e9962: Status 404 returned error can't find the container with id 36fb9e9b21ada0ba52e1b1fcce7e038446828d811aeb4202e5519cb5245e9962 Oct 30 20:50:41.705519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:41.705477 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:41.803678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:41.803431 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" event={"ID":"4a3b9107-6606-4a8c-806f-da7e770d09ae","Type":"ContainerStarted","Data":"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48"} Oct 30 20:50:41.803678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:41.803488 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" event={"ID":"4a3b9107-6606-4a8c-806f-da7e770d09ae","Type":"ContainerStarted","Data":"36fb9e9b21ada0ba52e1b1fcce7e038446828d811aeb4202e5519cb5245e9962"} Oct 30 20:50:41.831679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:41.831148 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podStartSLOduration=1.831127516 podStartE2EDuration="1.831127516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:41.829005696 +0000 UTC m=+655.187718443" watchObservedRunningTime="2026-10-30 20:50:41.831127516 +0000 UTC m=+655.189840273" Oct 30 20:50:42.607839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:42.607807 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:43.757054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:43.757018 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:43.770120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:43.770085 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:43.829761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:43.829733 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:50:49.706308 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:49.706263 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:49.720128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:49.720096 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:49.861184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:49.861152 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:50:50.608281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:50.608248 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:50.621792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:50.621767 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:50.867641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:50.867561 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:50:55.190359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:55.190319 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:50:56.618686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.618624 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:50:56.623909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.623884 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.629626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.629591 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"openshift-service-ca.crt\"" Oct 30 20:50:56.631021 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.630998 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"kube-root-ca.crt\"" Oct 30 20:50:56.631322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.631301 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"premium-w3-cb60a2-kserve-self-signed-certs\"" Oct 30 20:50:56.631504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.631489 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"default-dockercfg-kf8b2\"" Oct 30 20:50:56.633397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.633067 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:50:56.706355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706323 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.706565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706380 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.706620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706577 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.706701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706640 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.706752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706709 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5v4m\" (UniqueName: \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.706799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.706750 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809431 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809522 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809554 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5v4m\" (UniqueName: \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809605 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809690 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.809935 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.809751 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.811744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.810265 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.811744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.811411 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.811744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.811702 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.814193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.814168 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.814627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.814604 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.826951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.826918 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5v4m\" (UniqueName: \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\") pod \"premium-w3-cb60a2-kserve-74d58ccdc-g958k\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:56.939774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:56.939690 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:50:57.236023 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:57.235990 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:50:57.237953 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:50:57.237920 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22db12c6_81be_4871_829b_111f9ec4619f.slice/crio-5a1e6cacbbf6bdbb9f2a51ae2f0565c354618c4786160576fffe6e55b00a8e65 WatchSource:0}: Error finding container 5a1e6cacbbf6bdbb9f2a51ae2f0565c354618c4786160576fffe6e55b00a8e65: Status 404 returned error can't find the container with id 5a1e6cacbbf6bdbb9f2a51ae2f0565c354618c4786160576fffe6e55b00a8e65 Oct 30 20:50:57.892589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:57.892545 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" event={"ID":"22db12c6-81be-4871-829b-111f9ec4619f","Type":"ContainerStarted","Data":"b1ab7cca90762739734527567c0c5fa6413e8b22e517e790158df7118f004f99"} Oct 30 20:50:57.893132 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:57.892601 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" event={"ID":"22db12c6-81be-4871-829b-111f9ec4619f","Type":"ContainerStarted","Data":"5a1e6cacbbf6bdbb9f2a51ae2f0565c354618c4786160576fffe6e55b00a8e65"} Oct 30 20:50:57.914927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:57.914873 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podStartSLOduration=1.9148567380000001 podStartE2EDuration="1.914856738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:57.913539099 +0000 UTC m=+671.272251851" watchObservedRunningTime="2026-10-30 20:50:57.914856738 +0000 UTC m=+671.273569481" Oct 30 20:50:57.940820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:50:57.940787 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:51:00.925968 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:51:00.925928 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:51:01.474607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.474569 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:51:01.491990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.491957 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.492349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.492324 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:51:01.502808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.502776 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ad7ff0-openshift-default-dockercfg-pmg7l\"" Oct 30 20:51:01.564259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564221 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564263 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564287 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564374 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564393 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdpj\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564421 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564438 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564543 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.564717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.564601 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.665973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.665939 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.665973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.665980 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666033 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666092 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666122 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666275 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666343 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666376 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdpj\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666428 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666493 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666580 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666714 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666796 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.666955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.666934 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.668695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.668646 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.669468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.669437 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.688426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.688058 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.688426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.688055 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdpj\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:01.822076 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:01.822040 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:02.115040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.113209 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:51:02.119237 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:02.119182 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5d6f5e_154c_44ac_b6ac_4547a612f485.slice/crio-0d80a3d6fa1a15f3f46b833d85a5a45daeaf2d46b4a1a6ea770fcd3452e80f7a WatchSource:0}: Error finding container 0d80a3d6fa1a15f3f46b833d85a5a45daeaf2d46b4a1a6ea770fcd3452e80f7a: Status 404 returned error can't find the container with id 0d80a3d6fa1a15f3f46b833d85a5a45daeaf2d46b4a1a6ea770fcd3452e80f7a Oct 30 20:51:02.124917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.124404 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:02.124917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.124511 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:02.124917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.124551 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:02.926990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.926957 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" event={"ID":"be5d6f5e-154c-44ac-b6ac-4547a612f485","Type":"ContainerStarted","Data":"e05ea7a419f7ecc2481e96f7ddb4d07e2d47c25e995ee88db511115740685f49"} Oct 30 20:51:02.927185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.927015 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" event={"ID":"be5d6f5e-154c-44ac-b6ac-4547a612f485","Type":"ContainerStarted","Data":"0d80a3d6fa1a15f3f46b833d85a5a45daeaf2d46b4a1a6ea770fcd3452e80f7a"} Oct 30 20:51:02.957016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:02.956955 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podStartSLOduration=1.95693591 podStartE2EDuration="1.95693591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:02.95600403 +0000 UTC m=+676.314716788" watchObservedRunningTime="2026-10-30 20:51:02.95693591 +0000 UTC m=+676.315648658" Oct 30 20:51:03.822609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:03.822578 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:03.824151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:03.824126 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:03.824277 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:03.824172 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:04.824987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:04.824955 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:04.825449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:04.825426 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:05.827580 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:05.827096 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:05.827580 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:05.827164 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:06.822914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:06.822878 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:06.823196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:06.823165 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:06.940157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:06.940122 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:51:06.953135 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:06.953101 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:51:06.971418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:06.971390 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:51:07.822847 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:07.822810 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:07.823106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:07.823078 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:08.822597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:08.822563 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:51:08.823034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:08.822618 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:51:09.842792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:09.842760 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:09.843858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:09.843829 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:09.844996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:09.844960 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:51:16.719497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.719455 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:51:16.727285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.727255 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.736718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.736690 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:51:16.737520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.737500 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-db534b-openshift-default-dockercfg-4psrd\"" Oct 30 20:51:16.868854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.868816 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.868883 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.868984 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869049 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869084 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869190 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869162 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869190 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869179 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869228 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6fn\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.869302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.869282 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970554 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970605 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970646 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6fn\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970702 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970784 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970923 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970937 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.970974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.970972 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971025 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971049 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971393 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971357 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971384 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971410 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.971448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.971419 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.973467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.973436 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.973919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.973893 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.984539 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.984494 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:16.984707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:16.984599 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6fn\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:17.044342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.044303 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:17.281679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.281436 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:51:17.348877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.348846 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:51:17.350990 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:17.350945 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c96e45a_2a89_44b8_84bd_1fd0cddf5e5e.slice/crio-e940ef087af90775357da78202ed37489abbb2d0ebf1ced931ac5941b2d35ff3 WatchSource:0}: Error finding container e940ef087af90775357da78202ed37489abbb2d0ebf1ced931ac5941b2d35ff3: Status 404 returned error can't find the container with id e940ef087af90775357da78202ed37489abbb2d0ebf1ced931ac5941b2d35ff3 Oct 30 20:51:17.354205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.354162 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:17.354292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.354259 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:17.354334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:17.354294 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:18.054704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.050194 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" event={"ID":"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e","Type":"ContainerStarted","Data":"366cc0c898487958188a5252c4f9ab368d4fb7a8ab3c863bc3650c92a9140872"} Oct 30 20:51:18.054704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.050279 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" event={"ID":"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e","Type":"ContainerStarted","Data":"e940ef087af90775357da78202ed37489abbb2d0ebf1ced931ac5941b2d35ff3"} Oct 30 20:51:18.087687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.086336 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podStartSLOduration=2.086311703 podStartE2EDuration="2.086311703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:18.083293633 +0000 UTC m=+691.442006382" watchObservedRunningTime="2026-10-30 20:51:18.086311703 +0000 UTC m=+691.445024451" Oct 30 20:51:18.296679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.295641 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl"] Oct 30 20:51:18.304796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.304020 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.304796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.304396 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl"] Oct 30 20:51:18.308179 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.308152 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:18.308307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.308186 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:51:18.308372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.308355 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:51:18.308372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.308363 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:18.403257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.403218 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.403257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.403262 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.403501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.403372 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnlp\" (UniqueName: \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.504640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.504600 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.504858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.504645 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.504858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.504765 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnlp\" (UniqueName: \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.505307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.505280 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.507319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.507285 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.523102 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.523066 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnlp\" (UniqueName: \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\") pod \"maas-api-revoke-sub-5dae7b1d-rhfgl\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.628359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.626949 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:18.941669 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:18.941630 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl"] Oct 30 20:51:18.944515 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:18.944484 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1b51ad_6177_405d_ac18_b94e2da9ff90.slice/crio-1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db WatchSource:0}: Error finding container 1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db: Status 404 returned error can't find the container with id 1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db Oct 30 20:51:19.045263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:19.045229 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:19.046931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:19.046890 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:51:19.047086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:19.046960 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:51:19.057589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:19.057556 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" event={"ID":"3c1b51ad-6177-405d-ac18-b94e2da9ff90","Type":"ContainerStarted","Data":"1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db"} Oct 30 20:51:20.045199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:20.045141 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:51:20.045497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:20.045237 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:51:21.044800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:21.044761 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:51:21.045262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:21.044821 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:51:22.045299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:22.045254 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:51:22.045830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:22.045332 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:51:23.045402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:23.045363 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:51:23.045844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:23.045427 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:51:23.128111 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:23.128071 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" event={"ID":"3c1b51ad-6177-405d-ac18-b94e2da9ff90","Type":"ContainerStarted","Data":"1126bfb9b142589299b991b4c9069a4fc54e943a1fdf3838d4391ce4d6482dfe"} Oct 30 20:51:23.148922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:23.148865 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" podStartSLOduration=1.626731294 podStartE2EDuration="5.148850499s" totalImagesPullingTime="3.522119205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:23.147308275 +0000 UTC m=+696.506021028" watchObservedRunningTime="2026-10-30 20:51:23.148850499 +0000 UTC m=+696.507563248" Oct 30 20:51:24.139550 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:24.139462 5855 generic.go:356] "Generic (PLEG): container finished" podID="3c1b51ad-6177-405d-ac18-b94e2da9ff90" containerID="1126bfb9b142589299b991b4c9069a4fc54e943a1fdf3838d4391ce4d6482dfe" exitCode=0 Oct 30 20:51:24.140249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:24.139556 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" event={"ID":"3c1b51ad-6177-405d-ac18-b94e2da9ff90","Type":"ContainerDied","Data":"1126bfb9b142589299b991b4c9069a4fc54e943a1fdf3838d4391ce4d6482dfe"} Oct 30 20:51:24.866052 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:24.866002 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:24.866754 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:24.866452 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:24.867386 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:24.867365 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:51:25.282217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.282184 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:25.286228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.286204 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\" (UniqueName: \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\") pod \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " Oct 30 20:51:25.286352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.286289 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\") pod \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " Oct 30 20:51:25.286352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.286331 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\") pod \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\" (UID: \"3c1b51ad-6177-405d-ac18-b94e2da9ff90\") " Oct 30 20:51:25.286807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.286778 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca" pod "3c1b51ad-6177-405d-ac18-b94e2da9ff90" (UID: "3c1b51ad-6177-405d-ac18-b94e2da9ff90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:25.288706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.288676 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca" pod "3c1b51ad-6177-405d-ac18-b94e2da9ff90" (UID: "3c1b51ad-6177-405d-ac18-b94e2da9ff90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:25.288791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.288726 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp" pod "3c1b51ad-6177-405d-ac18-b94e2da9ff90" (UID: "3c1b51ad-6177-405d-ac18-b94e2da9ff90"). InnerVolumeSpecName "kube-api-access-mpnlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:25.388016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.387978 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpnlp\" (UniqueName: \"kubernetes.io/projected/3c1b51ad-6177-405d-ac18-b94e2da9ff90-kube-api-access-mpnlp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:25.388016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.388015 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:25.388225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:25.388028 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c1b51ad-6177-405d-ac18-b94e2da9ff90-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:26.150499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:26.150468 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" event={"ID":"3c1b51ad-6177-405d-ac18-b94e2da9ff90","Type":"ContainerDied","Data":"1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db"} Oct 30 20:51:26.150499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:26.150496 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl" Oct 30 20:51:26.150736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:26.150522 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ddf60194f3d49f93ecfde24146b57889084ef2ca8610ae706572a0f63b4e0db" Oct 30 20:51:30.925704 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:51:30.925617 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:51:32.567228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.567191 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4"] Oct 30 20:51:32.567788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.567621 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3c1b51ad-6177-405d-ac18-b94e2da9ff90" containerName="revoke-keys" Oct 30 20:51:32.567788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.567632 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1b51ad-6177-405d-ac18-b94e2da9ff90" containerName="revoke-keys" Oct 30 20:51:32.567788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.567760 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1b51ad-6177-405d-ac18-b94e2da9ff90" containerName="revoke-keys" Oct 30 20:51:32.571049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.571026 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.578264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.578235 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:32.578861 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.578638 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:51:32.578861 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.578795 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:32.579467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.579443 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:51:32.582020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.581999 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4"] Oct 30 20:51:32.666116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.666079 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.666289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.666147 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.666332 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.666279 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crc64\" (UniqueName: \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.767634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.767598 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.767883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.767671 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.767883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.767715 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crc64\" (UniqueName: \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.768309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.768285 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.770365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.770340 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.782874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.782844 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crc64\" (UniqueName: \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\") pod \"maas-api-revoke-sub-07c9e285-t8fq4\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:32.889469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:32.889377 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:33.174592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:33.174566 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4"] Oct 30 20:51:33.176880 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:33.176846 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1480c7a_756b_4550_b917_e5ad20542517.slice/crio-1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604 WatchSource:0}: Error finding container 1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604: Status 404 returned error can't find the container with id 1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604 Oct 30 20:51:33.199606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:33.199577 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" event={"ID":"f1480c7a-756b-4550-b917-e5ad20542517","Type":"ContainerStarted","Data":"1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604"} Oct 30 20:51:34.207442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:34.207398 5855 generic.go:356] "Generic (PLEG): container finished" podID="f1480c7a-756b-4550-b917-e5ad20542517" containerID="a53806d852cead9fb76f3b520f7cdd187befa0352d7e253c6d30bb47c0cdd792" exitCode=0 Oct 30 20:51:34.207891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:34.207491 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" event={"ID":"f1480c7a-756b-4550-b917-e5ad20542517","Type":"ContainerDied","Data":"a53806d852cead9fb76f3b520f7cdd187befa0352d7e253c6d30bb47c0cdd792"} Oct 30 20:51:35.355066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.355038 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:35.400319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.400273 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\" (UniqueName: \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\") pod \"f1480c7a-756b-4550-b917-e5ad20542517\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " Oct 30 20:51:35.400499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.400339 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\") pod \"f1480c7a-756b-4550-b917-e5ad20542517\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " Oct 30 20:51:35.400499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.400465 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\") pod \"f1480c7a-756b-4550-b917-e5ad20542517\" (UID: \"f1480c7a-756b-4550-b917-e5ad20542517\") " Oct 30 20:51:35.400753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.400721 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca" pod "f1480c7a-756b-4550-b917-e5ad20542517" (UID: "f1480c7a-756b-4550-b917-e5ad20542517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:35.400920 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.400902 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:35.402693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.402648 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64" pod "f1480c7a-756b-4550-b917-e5ad20542517" (UID: "f1480c7a-756b-4550-b917-e5ad20542517"). InnerVolumeSpecName "kube-api-access-crc64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:35.403072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.403051 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca" pod "f1480c7a-756b-4550-b917-e5ad20542517" (UID: "f1480c7a-756b-4550-b917-e5ad20542517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:35.501986 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.501892 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1480c7a-756b-4550-b917-e5ad20542517-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:35.501986 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:35.501932 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crc64\" (UniqueName: \"kubernetes.io/projected/f1480c7a-756b-4550-b917-e5ad20542517-kube-api-access-crc64\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:36.220812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:36.220775 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" Oct 30 20:51:36.220994 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:36.220783 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4" event={"ID":"f1480c7a-756b-4550-b917-e5ad20542517","Type":"ContainerDied","Data":"1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604"} Oct 30 20:51:36.220994 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:36.220909 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d26efa84a2994e569ac279257f3441f5674217cb9b9e2465a66cfc7294d8604" Oct 30 20:51:37.076354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:37.076318 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:51:45.558129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.558094 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf"] Oct 30 20:51:45.559240 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.559213 5855 cpu_manager.go:414] "containerMap: removing container" podUID="f1480c7a-756b-4550-b917-e5ad20542517" containerName="revoke-keys" Oct 30 20:51:45.559240 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.559239 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1480c7a-756b-4550-b917-e5ad20542517" containerName="revoke-keys" Oct 30 20:51:45.559421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.559356 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1480c7a-756b-4550-b917-e5ad20542517" containerName="revoke-keys" Oct 30 20:51:45.563144 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.563127 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.568894 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.568868 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:45.569034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.568956 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:51:45.569034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.569003 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:45.569570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.569545 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:51:45.570952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.570930 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf"] Oct 30 20:51:45.622115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.622079 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwfz\" (UniqueName: \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.622282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.622151 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.622282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.622265 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.723051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.722997 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwfz\" (UniqueName: \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.723242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.723114 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.723242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.723172 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.724072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.724045 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.725875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.725846 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.736896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.736860 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwfz\" (UniqueName: \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\") pod \"maas-api-revoke-sub-e7c3e573-x4wnf\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:45.879259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:45.879170 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:46.014396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.014363 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x"] Oct 30 20:51:46.018302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.018283 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.026884 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.026522 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x"] Oct 30 20:51:46.126966 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.126925 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.127169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.126985 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.127169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.127062 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6nz\" (UniqueName: \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.159437 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.159403 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf"] Oct 30 20:51:46.161906 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:46.161856 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4b1c99_451c_4577_8416_42082f152425.slice/crio-faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62 WatchSource:0}: Error finding container faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62: Status 404 returned error can't find the container with id faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62 Oct 30 20:51:46.228097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.228059 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.228309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.228125 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.228309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.228157 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6nz\" (UniqueName: \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.228862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.228842 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.230811 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.230781 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.240811 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.240787 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6nz\" (UniqueName: \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\") pod \"maas-api-revoke-sub-0edf61d9-28z7x\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.273149 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.273103 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" event={"ID":"ae4b1c99-451c-4577-8416-42082f152425","Type":"ContainerStarted","Data":"faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62"} Oct 30 20:51:46.281878 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.281849 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 20:51:46.286924 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.286904 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.299152 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.299125 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f2ed1e-openshift-default-dockercfg-g7p8g\"" Oct 30 20:51:46.301985 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.301958 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 20:51:46.334188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.334152 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434258 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434336 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxjt\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434394 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434453 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434480 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434526 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434582 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434608 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.440745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.434696 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535683 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535760 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535786 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535830 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535885 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.535941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535908 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536195 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.535974 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536195 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536033 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536195 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536084 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxjt\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536194 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536218 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536285 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536580 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536556 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.536697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.536620 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.538609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.538589 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.539054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.539035 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.549226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.549203 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.549346 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.549298 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxjt\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.601234 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.601205 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:46.617767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.617743 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x"] Oct 30 20:51:46.619704 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:46.619673 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03bea02_9a84_47a8_894a_62edc8a9dcf8.slice/crio-66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab WatchSource:0}: Error finding container 66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab: Status 404 returned error can't find the container with id 66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab Oct 30 20:51:46.896004 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.895975 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 20:51:46.897390 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:46.897361 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f8ac57_eaae_4e5d_93d2_499338eddf24.slice/crio-a8510da3c6dea9a6a6f47be6718527c3f49bad65f22ce215b15292f471ce1564 WatchSource:0}: Error finding container a8510da3c6dea9a6a6f47be6718527c3f49bad65f22ce215b15292f471ce1564: Status 404 returned error can't find the container with id a8510da3c6dea9a6a6f47be6718527c3f49bad65f22ce215b15292f471ce1564 Oct 30 20:51:46.900143 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.900108 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:46.900263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.900186 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:46.900263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:46.900231 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:51:47.281293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.281255 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" event={"ID":"a03bea02-9a84-47a8-894a-62edc8a9dcf8","Type":"ContainerStarted","Data":"977ede348b53c531b894c0c7150ba62e82e258f1a00fe27ea88620e2bbe8ce8b"} Oct 30 20:51:47.281501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.281321 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" event={"ID":"a03bea02-9a84-47a8-894a-62edc8a9dcf8","Type":"ContainerStarted","Data":"66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab"} Oct 30 20:51:47.283770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.283731 5855 generic.go:356] "Generic (PLEG): container finished" podID="ae4b1c99-451c-4577-8416-42082f152425" containerID="ed797f2c9ec10716f5516770eabdb32b733c08ff4168cf04c7c54b91a7f4706f" exitCode=0 Oct 30 20:51:47.283921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.283808 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" event={"ID":"ae4b1c99-451c-4577-8416-42082f152425","Type":"ContainerDied","Data":"ed797f2c9ec10716f5516770eabdb32b733c08ff4168cf04c7c54b91a7f4706f"} Oct 30 20:51:47.285836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.285800 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" event={"ID":"30f8ac57-eaae-4e5d-93d2-499338eddf24","Type":"ContainerStarted","Data":"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7"} Oct 30 20:51:47.285951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.285853 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" event={"ID":"30f8ac57-eaae-4e5d-93d2-499338eddf24","Type":"ContainerStarted","Data":"a8510da3c6dea9a6a6f47be6718527c3f49bad65f22ce215b15292f471ce1564"} Oct 30 20:51:47.301887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.301820 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" podStartSLOduration=2.301800178 podStartE2EDuration="2.301800178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:47.299997979 +0000 UTC m=+720.658710726" watchObservedRunningTime="2026-10-30 20:51:47.301800178 +0000 UTC m=+720.660512925" Oct 30 20:51:47.325728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.325646 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podStartSLOduration=1.325629411 podStartE2EDuration="1.325629411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:47.323358525 +0000 UTC m=+720.682071273" watchObservedRunningTime="2026-10-30 20:51:47.325629411 +0000 UTC m=+720.684342158" Oct 30 20:51:47.604885 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.603635 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:47.604885 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.603825 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:47.604885 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:47.603874 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:48.298057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.298017 5855 generic.go:356] "Generic (PLEG): container finished" podID="a03bea02-9a84-47a8-894a-62edc8a9dcf8" containerID="977ede348b53c531b894c0c7150ba62e82e258f1a00fe27ea88620e2bbe8ce8b" exitCode=0 Oct 30 20:51:48.305260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.305176 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" event={"ID":"a03bea02-9a84-47a8-894a-62edc8a9dcf8","Type":"ContainerDied","Data":"977ede348b53c531b894c0c7150ba62e82e258f1a00fe27ea88620e2bbe8ce8b"} Oct 30 20:51:48.513432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.513400 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:48.559289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.559199 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\" (UniqueName: \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\") pod \"ae4b1c99-451c-4577-8416-42082f152425\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " Oct 30 20:51:48.559289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.559252 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\") pod \"ae4b1c99-451c-4577-8416-42082f152425\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " Oct 30 20:51:48.559516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.559391 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\") pod \"ae4b1c99-451c-4577-8416-42082f152425\" (UID: \"ae4b1c99-451c-4577-8416-42082f152425\") " Oct 30 20:51:48.559925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.559890 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca" pod "ae4b1c99-451c-4577-8416-42082f152425" (UID: "ae4b1c99-451c-4577-8416-42082f152425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:48.561605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.561574 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca" pod "ae4b1c99-451c-4577-8416-42082f152425" (UID: "ae4b1c99-451c-4577-8416-42082f152425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:48.561993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.561970 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz" pod "ae4b1c99-451c-4577-8416-42082f152425" (UID: "ae4b1c99-451c-4577-8416-42082f152425"). InnerVolumeSpecName "kube-api-access-dzwfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:48.604683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.604626 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:48.604942 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.604713 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:48.661129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.661085 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwfz\" (UniqueName: \"kubernetes.io/projected/ae4b1c99-451c-4577-8416-42082f152425-kube-api-access-dzwfz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:48.661129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.661119 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4b1c99-451c-4577-8416-42082f152425-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:48.661129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:48.661130 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4b1c99-451c-4577-8416-42082f152425-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:49.310944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.310904 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" Oct 30 20:51:49.311116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.310954 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf" event={"ID":"ae4b1c99-451c-4577-8416-42082f152425","Type":"ContainerDied","Data":"faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62"} Oct 30 20:51:49.311116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.311031 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faf6b209e3774437da128da565de6a5023192bca1ee3e26e636e6af7261a4f62" Oct 30 20:51:49.519648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.519615 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:49.571466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.571382 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\") pod \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " Oct 30 20:51:49.571613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.571518 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\") pod \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " Oct 30 20:51:49.571613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.571568 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\" (UniqueName: \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\") pod \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\" (UID: \"a03bea02-9a84-47a8-894a-62edc8a9dcf8\") " Oct 30 20:51:49.571980 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.571943 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca" pod "a03bea02-9a84-47a8-894a-62edc8a9dcf8" (UID: "a03bea02-9a84-47a8-894a-62edc8a9dcf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:49.572125 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.572066 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:49.605967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.602339 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:49.605967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.602388 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:49.605967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.603301 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz" pod "a03bea02-9a84-47a8-894a-62edc8a9dcf8" (UID: "a03bea02-9a84-47a8-894a-62edc8a9dcf8"). InnerVolumeSpecName "kube-api-access-qc6nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:49.610070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.610036 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca" pod "a03bea02-9a84-47a8-894a-62edc8a9dcf8" (UID: "a03bea02-9a84-47a8-894a-62edc8a9dcf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:49.690684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.673072 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6nz\" (UniqueName: \"kubernetes.io/projected/a03bea02-9a84-47a8-894a-62edc8a9dcf8-kube-api-access-qc6nz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:49.690684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:49.673106 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03bea02-9a84-47a8-894a-62edc8a9dcf8-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:50.319065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:50.319025 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" event={"ID":"a03bea02-9a84-47a8-894a-62edc8a9dcf8","Type":"ContainerDied","Data":"66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab"} Oct 30 20:51:50.319226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:50.319091 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66506b7f79847cd9546540babf243ba1e4b645d05859a56ca1c65f4318fe8fab" Oct 30 20:51:50.319226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:50.319046 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x" Oct 30 20:51:50.602171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:50.602077 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:50.602464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:50.602422 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:51.602165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:51.602119 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:51.602890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:51.602189 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:52.605704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:52.605491 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 20:51:52.605704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:52.605619 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 20:51:53.864302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:53.864268 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:53.865046 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:53.864808 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:53.865552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:53.865529 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 20:51:55.769551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.769473 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq"] Oct 30 20:51:55.769970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.769898 5855 cpu_manager.go:414] "containerMap: removing container" podUID="a03bea02-9a84-47a8-894a-62edc8a9dcf8" containerName="revoke-keys" Oct 30 20:51:55.769970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.769911 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03bea02-9a84-47a8-894a-62edc8a9dcf8" containerName="revoke-keys" Oct 30 20:51:55.769970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.769924 5855 cpu_manager.go:414] "containerMap: removing container" podUID="ae4b1c99-451c-4577-8416-42082f152425" containerName="revoke-keys" Oct 30 20:51:55.769970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.769929 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4b1c99-451c-4577-8416-42082f152425" containerName="revoke-keys" Oct 30 20:51:55.770101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.770004 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03bea02-9a84-47a8-894a-62edc8a9dcf8" containerName="revoke-keys" Oct 30 20:51:55.770101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.770011 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4b1c99-451c-4577-8416-42082f152425" containerName="revoke-keys" Oct 30 20:51:55.776401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.776378 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.782116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.782079 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq"] Oct 30 20:51:55.784807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.783826 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:55.784807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.783876 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:51:55.784807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.783941 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:55.784807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.783942 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:51:55.890532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.890486 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.890738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.890603 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.890738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.890646 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9kf\" (UniqueName: \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.992349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.992312 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.992522 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.992363 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.992522 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.992481 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9kf\" (UniqueName: \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.993314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.993291 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:55.995121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:55.995091 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:56.023504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:56.023435 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9kf\" (UniqueName: \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\") pod \"maas-api-revoke-sub-5f218d58-jbwmq\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:56.099893 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:56.099854 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:56.384409 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:56.384377 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq"] Oct 30 20:51:56.385830 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:56.385790 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab33f8a3_7a8c_4565_861f_be4add5ba8d9.slice/crio-5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491 WatchSource:0}: Error finding container 5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491: Status 404 returned error can't find the container with id 5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491 Oct 30 20:51:57.365482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:57.365440 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" event={"ID":"ab33f8a3-7a8c-4565-861f-be4add5ba8d9","Type":"ContainerStarted","Data":"f6b042b6d62f7770cb9976ca7300b64d0dfce5ad58c87513673c72395a4f31ef"} Oct 30 20:51:57.365949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:57.365512 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" event={"ID":"ab33f8a3-7a8c-4565-861f-be4add5ba8d9","Type":"ContainerStarted","Data":"5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491"} Oct 30 20:51:57.391922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:57.391875 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" podStartSLOduration=2.391862078 podStartE2EDuration="2.391862078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:57.390024169 +0000 UTC m=+730.748736915" watchObservedRunningTime="2026-10-30 20:51:57.391862078 +0000 UTC m=+730.750574824" Oct 30 20:51:58.372044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:58.372007 5855 generic.go:356] "Generic (PLEG): container finished" podID="ab33f8a3-7a8c-4565-861f-be4add5ba8d9" containerID="f6b042b6d62f7770cb9976ca7300b64d0dfce5ad58c87513673c72395a4f31ef" exitCode=0 Oct 30 20:51:58.372466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:58.372073 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" event={"ID":"ab33f8a3-7a8c-4565-861f-be4add5ba8d9","Type":"ContainerDied","Data":"f6b042b6d62f7770cb9976ca7300b64d0dfce5ad58c87513673c72395a4f31ef"} Oct 30 20:51:59.188782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.188739 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt"] Oct 30 20:51:59.192596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.192573 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.202788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.202764 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt"] Oct 30 20:51:59.336154 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.336113 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpwc\" (UniqueName: \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.336349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.336255 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.336395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.336380 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.437326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.437276 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.437892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.437432 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.437892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.437487 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpwc\" (UniqueName: \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.438505 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.438382 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.446319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.446228 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.451087 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.451036 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpwc\" (UniqueName: \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\") pod \"maas-api-revoke-sub-7b230c2d-sfwjt\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.510555 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.510518 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:51:59.559421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.559390 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:51:59.627439 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.627398 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq"] Oct 30 20:51:59.628101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.628078 5855 cpu_manager.go:414] "containerMap: removing container" podUID="ab33f8a3-7a8c-4565-861f-be4add5ba8d9" containerName="revoke-keys" Oct 30 20:51:59.628161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.628105 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab33f8a3-7a8c-4565-861f-be4add5ba8d9" containerName="revoke-keys" Oct 30 20:51:59.628249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.628236 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab33f8a3-7a8c-4565-861f-be4add5ba8d9" containerName="revoke-keys" Oct 30 20:51:59.631736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.631712 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.638464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.638426 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq"] Oct 30 20:51:59.740716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.740624 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\") pod \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " Oct 30 20:51:59.740716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.740711 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\") pod \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " Oct 30 20:51:59.740917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.740783 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\" (UniqueName: \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\") pod \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\" (UID: \"ab33f8a3-7a8c-4565-861f-be4add5ba8d9\") " Oct 30 20:51:59.740966 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.740945 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.741163 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.741115 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjnh\" (UniqueName: \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.741163 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.741127 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca" pod "ab33f8a3-7a8c-4565-861f-be4add5ba8d9" (UID: "ab33f8a3-7a8c-4565-861f-be4add5ba8d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:59.741303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.741245 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.741444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.741413 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:59.743033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.743011 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf" pod "ab33f8a3-7a8c-4565-861f-be4add5ba8d9" (UID: "ab33f8a3-7a8c-4565-861f-be4add5ba8d9"). InnerVolumeSpecName "kube-api-access-9s9kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:59.743226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.743212 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca" pod "ab33f8a3-7a8c-4565-861f-be4add5ba8d9" (UID: "ab33f8a3-7a8c-4565-861f-be4add5ba8d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:59.794311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.794265 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt"] Oct 30 20:51:59.796092 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:51:59.796062 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f02a5f_958a_4ab6_a452_3b4d06d440bd.slice/crio-d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a WatchSource:0}: Error finding container d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a: Status 404 returned error can't find the container with id d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a Oct 30 20:51:59.842459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.842423 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjnh\" (UniqueName: \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.842640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.842489 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.842640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.842543 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.842640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.842610 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:59.842640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.842620 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9kf\" (UniqueName: \"kubernetes.io/projected/ab33f8a3-7a8c-4565-861f-be4add5ba8d9-kube-api-access-9s9kf\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:59.843251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.843232 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.844999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.844982 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.856072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.856035 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjnh\" (UniqueName: \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\") pod \"maas-api-revoke-sub-59d24306-4hxqq\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:51:59.954808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:51:59.954776 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:52:00.240322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.240297 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq"] Oct 30 20:52:00.242184 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:00.242156 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f05e3f_15c9_49c0_bb76_5135ce9fbb81.slice/crio-025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8 WatchSource:0}: Error finding container 025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8: Status 404 returned error can't find the container with id 025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8 Oct 30 20:52:00.385482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.385453 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" Oct 30 20:52:00.385742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.385451 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq" event={"ID":"ab33f8a3-7a8c-4565-861f-be4add5ba8d9","Type":"ContainerDied","Data":"5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491"} Oct 30 20:52:00.385742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.385624 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b548ee800972368af87361447a359af292f828bd4f6d117799907b2c9e39491" Oct 30 20:52:00.386968 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.386937 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" event={"ID":"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81","Type":"ContainerStarted","Data":"025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8"} Oct 30 20:52:00.388795 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.388772 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" event={"ID":"82f02a5f-958a-4ab6-a452-3b4d06d440bd","Type":"ContainerStarted","Data":"84da9ca87958439ed25938ad2eeb4fdff78ef02a6c72c33a61503d68ffe70c21"} Oct 30 20:52:00.388933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.388811 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" event={"ID":"82f02a5f-958a-4ab6-a452-3b4d06d440bd","Type":"ContainerStarted","Data":"d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a"} Oct 30 20:52:00.410343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:00.410292 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" podStartSLOduration=1.410277594 podStartE2EDuration="1.410277594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:00.407254571 +0000 UTC m=+733.765967321" watchObservedRunningTime="2026-10-30 20:52:00.410277594 +0000 UTC m=+733.768990340" Oct 30 20:52:00.925746 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:52:00.925641 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:52:01.396636 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.396602 5855 generic.go:356] "Generic (PLEG): container finished" podID="a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" containerID="214ddfaa6548246c91f493cbe96ae28f3ed8cd39161a67734315eaf916c723df" exitCode=0 Oct 30 20:52:01.396839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.396692 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" event={"ID":"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81","Type":"ContainerDied","Data":"214ddfaa6548246c91f493cbe96ae28f3ed8cd39161a67734315eaf916c723df"} Oct 30 20:52:01.398348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.398322 5855 generic.go:356] "Generic (PLEG): container finished" podID="82f02a5f-958a-4ab6-a452-3b4d06d440bd" containerID="84da9ca87958439ed25938ad2eeb4fdff78ef02a6c72c33a61503d68ffe70c21" exitCode=0 Oct 30 20:52:01.398468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.398375 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" event={"ID":"82f02a5f-958a-4ab6-a452-3b4d06d440bd","Type":"ContainerDied","Data":"84da9ca87958439ed25938ad2eeb4fdff78ef02a6c72c33a61503d68ffe70c21"} Oct 30 20:52:01.634496 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.634458 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 20:52:01.638315 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.638292 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.649882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.649843 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 20:52:01.664640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.664607 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b8aee1-openshift-default-dockercfg-trpph\"" Oct 30 20:52:01.772636 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.772604 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.772859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.772641 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.772859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.772747 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.772978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.772943 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.773022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.772983 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.773096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.773081 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.773131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.773115 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.773394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.773178 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrzw\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.773394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.773204 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874588 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874556 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874818 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874597 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874818 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874767 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrzw\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874836 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874920 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.874958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.874944 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875058 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875206 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875296 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875246 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875296 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875273 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875726 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875879 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875833 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.875956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.875923 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.876336 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.876316 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.877605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.877578 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.877859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.877840 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.889719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.889634 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrzw\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.889719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.889647 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:01.954301 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:01.954260 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:02.272858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.272829 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 20:52:02.274199 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:02.274171 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f57bb2_f46f_42f0_8b48_5712dcb3eae2.slice/crio-ba7c003f74fdd662425aeb4104bfad1ef4be9d20dd5ee9a3471c938d4ccd0fdd WatchSource:0}: Error finding container ba7c003f74fdd662425aeb4104bfad1ef4be9d20dd5ee9a3471c938d4ccd0fdd: Status 404 returned error can't find the container with id ba7c003f74fdd662425aeb4104bfad1ef4be9d20dd5ee9a3471c938d4ccd0fdd Oct 30 20:52:02.276645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.276613 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:02.276768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.276729 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:02.276806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.276768 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:02.408210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.408150 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" event={"ID":"22f57bb2-f46f-42f0-8b48-5712dcb3eae2","Type":"ContainerStarted","Data":"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a"} Oct 30 20:52:02.408210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.408212 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" event={"ID":"22f57bb2-f46f-42f0-8b48-5712dcb3eae2","Type":"ContainerStarted","Data":"ba7c003f74fdd662425aeb4104bfad1ef4be9d20dd5ee9a3471c938d4ccd0fdd"} Oct 30 20:52:02.436107 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.436038 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podStartSLOduration=1.436015771 podStartE2EDuration="1.436015771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:02.431937027 +0000 UTC m=+735.790649774" watchObservedRunningTime="2026-10-30 20:52:02.436015771 +0000 UTC m=+735.794728518" Oct 30 20:52:02.584876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.584844 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:52:02.609562 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.609535 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:52:02.694975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.694941 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\" (UniqueName: \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\") pod \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " Oct 30 20:52:02.695168 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.695021 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\") pod \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " Oct 30 20:52:02.695168 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.695107 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\" (UniqueName: \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\") pod \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " Oct 30 20:52:02.695306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.695167 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\") pod \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " Oct 30 20:52:02.695306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.695242 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\") pod \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\" (UID: \"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81\") " Oct 30 20:52:02.695306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.695280 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\") pod \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\" (UID: \"82f02a5f-958a-4ab6-a452-3b4d06d440bd\") " Oct 30 20:52:02.696257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.696225 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca" pod "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" (UID: "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:02.696445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.696384 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca" pod "82f02a5f-958a-4ab6-a452-3b4d06d440bd" (UID: "82f02a5f-958a-4ab6-a452-3b4d06d440bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:02.698569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.698527 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh" pod "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" (UID: "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81"). InnerVolumeSpecName "kube-api-access-qrjnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:02.698569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.698575 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca" pod "82f02a5f-958a-4ab6-a452-3b4d06d440bd" (UID: "82f02a5f-958a-4ab6-a452-3b4d06d440bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:02.698872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.698647 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc" pod "82f02a5f-958a-4ab6-a452-3b4d06d440bd" (UID: "82f02a5f-958a-4ab6-a452-3b4d06d440bd"). InnerVolumeSpecName "kube-api-access-hnpwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:02.699053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.699032 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca" pod "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" (UID: "a1f05e3f-15c9-49c0-bb76-5135ce9fbb81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:02.798431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798393 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpwc\" (UniqueName: \"kubernetes.io/projected/82f02a5f-958a-4ab6-a452-3b4d06d440bd-kube-api-access-hnpwc\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.798591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798436 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.798591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798463 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.798591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798478 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.798591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798492 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjnh\" (UniqueName: \"kubernetes.io/projected/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81-kube-api-access-qrjnh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.798591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.798505 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f02a5f-958a-4ab6-a452-3b4d06d440bd-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.954569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:02.954535 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:03.415005 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.414977 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" Oct 30 20:52:03.415217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.415017 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq" event={"ID":"a1f05e3f-15c9-49c0-bb76-5135ce9fbb81","Type":"ContainerDied","Data":"025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8"} Oct 30 20:52:03.415217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.415072 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="025b4c0f50901af7b61d7641c3ff3017930a64c300367a8451516fd4ebda4bc8" Oct 30 20:52:03.416603 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.416582 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" Oct 30 20:52:03.416603 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.416591 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt" event={"ID":"82f02a5f-958a-4ab6-a452-3b4d06d440bd","Type":"ContainerDied","Data":"d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a"} Oct 30 20:52:03.416825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.416635 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d365d8c1cf58fa75540a7750f3b9c51a49d39b805bcd1f134dfc3f29c056753a" Oct 30 20:52:03.957188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.957152 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:03.957597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:03.957212 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:04.954812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:04.954783 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:04.954993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:04.954834 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:05.954593 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:05.954558 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:05.955079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:05.954630 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:05.983680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:05.980506 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:52:06.958193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:06.958155 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:06.958819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:06.958777 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:07.956128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:07.956082 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:07.956445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:07.956412 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:08.956135 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:08.956095 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:52:08.956800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:08.956763 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:52:10.499589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.499539 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s"] Oct 30 20:52:10.500270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500248 5855 cpu_manager.go:414] "containerMap: removing container" podUID="a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" containerName="revoke-keys" Oct 30 20:52:10.500392 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500270 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" containerName="revoke-keys" Oct 30 20:52:10.500392 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500302 5855 cpu_manager.go:414] "containerMap: removing container" podUID="82f02a5f-958a-4ab6-a452-3b4d06d440bd" containerName="revoke-keys" Oct 30 20:52:10.500392 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500311 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f02a5f-958a-4ab6-a452-3b4d06d440bd" containerName="revoke-keys" Oct 30 20:52:10.500560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500402 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" containerName="revoke-keys" Oct 30 20:52:10.500560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.500414 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f02a5f-958a-4ab6-a452-3b4d06d440bd" containerName="revoke-keys" Oct 30 20:52:10.505181 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.505158 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.514099 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.513167 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:10.514504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.513567 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:52:10.515043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.515025 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:52:10.515330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.515292 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:10.516917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.516876 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s"] Oct 30 20:52:10.583271 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.583235 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kx2g\" (UniqueName: \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.583559 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.583541 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.583822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.583804 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.685676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.685103 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.685676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.685276 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kx2g\" (UniqueName: \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.685676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.685375 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.688791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.686308 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.701683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.701122 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kx2g\" (UniqueName: \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.706257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.706221 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767543ca-jzn2s\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.830679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.830198 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9"] Oct 30 20:52:10.844416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.842269 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9"] Oct 30 20:52:10.844416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.842423 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.844416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.844107 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:10.895404 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.895362 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.895596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.895420 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwkq\" (UniqueName: \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.895702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.895618 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.956889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.956827 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:10.956976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.956913 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:10.998417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.998377 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.999677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.999511 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:10.999948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:10.999931 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:11.001252 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.001233 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwkq\" (UniqueName: \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:11.004188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.004144 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:11.012334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.010939 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:11.012334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.012260 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:11.012959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.012908 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 20:52:11.019044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.019022 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwkq\" (UniqueName: \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\") pod \"maas-api-revoke-sub-8d17cf35-28md9\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:11.175482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.175443 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s"] Oct 30 20:52:11.176761 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:11.176732 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae59a59e_0964_4bfa_8e6b_c498354d291a.slice/crio-f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280 WatchSource:0}: Error finding container f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280: Status 404 returned error can't find the container with id f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280 Oct 30 20:52:11.197681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.197641 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:11.477642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.477559 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" event={"ID":"ae59a59e-0964-4bfa-8e6b-c498354d291a","Type":"ContainerStarted","Data":"f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280"} Oct 30 20:52:11.487700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.487649 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9"] Oct 30 20:52:11.491637 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:11.491606 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621956fe_e412_4fe9_9844_bfd7993a31ca.slice/crio-df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7 WatchSource:0}: Error finding container df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7: Status 404 returned error can't find the container with id df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7 Oct 30 20:52:11.787855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.787771 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99"] Oct 30 20:52:11.792168 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.792139 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:11.802703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.802241 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99"] Oct 30 20:52:11.913582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.913546 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:11.913799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.913726 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:11.913799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:11.913783 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tkd\" (UniqueName: \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.014898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.014858 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tkd\" (UniqueName: \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.015093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.014926 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.015093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.015067 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.016073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.016037 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.018820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.017803 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.034496 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.034471 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tkd\" (UniqueName: \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\") pod \"maas-api-revoke-sub-dd58f743-xjm99\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.114479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.114440 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:12.402133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.402110 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99"] Oct 30 20:52:12.404531 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:12.404504 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43027be5_25ba_492f_9824_3a9ae85dab5b.slice/crio-00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd WatchSource:0}: Error finding container 00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd: Status 404 returned error can't find the container with id 00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd Oct 30 20:52:12.485624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.485588 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" event={"ID":"621956fe-e412-4fe9-9844-bfd7993a31ca","Type":"ContainerStarted","Data":"a901405d811c6aedc4b9ccbfd7f07e7736c1768000eec91fdbc61c6ebb194db6"} Oct 30 20:52:12.485814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.485669 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" event={"ID":"621956fe-e412-4fe9-9844-bfd7993a31ca","Type":"ContainerStarted","Data":"df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7"} Oct 30 20:52:12.487546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.487429 5855 generic.go:356] "Generic (PLEG): container finished" podID="ae59a59e-0964-4bfa-8e6b-c498354d291a" containerID="f3cd4faf89def1a096f0abaecf2636bc38dd8dc59c4cfbc54beb1e33f236ecb6" exitCode=0 Oct 30 20:52:12.487546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.487476 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" event={"ID":"ae59a59e-0964-4bfa-8e6b-c498354d291a","Type":"ContainerDied","Data":"f3cd4faf89def1a096f0abaecf2636bc38dd8dc59c4cfbc54beb1e33f236ecb6"} Oct 30 20:52:12.489250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.489222 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" event={"ID":"43027be5-25ba-492f-9824-3a9ae85dab5b","Type":"ContainerStarted","Data":"00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd"} Oct 30 20:52:12.510075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:12.510004 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" podStartSLOduration=2.509982402 podStartE2EDuration="2.509982402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:12.507762132 +0000 UTC m=+745.866474878" watchObservedRunningTime="2026-10-30 20:52:12.509982402 +0000 UTC m=+745.868695149" Oct 30 20:52:13.497116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.497080 5855 generic.go:356] "Generic (PLEG): container finished" podID="43027be5-25ba-492f-9824-3a9ae85dab5b" containerID="84951baa67eedc31b8095ecf087599dfbaac7a7ffa9266905a0b43d3c9054995" exitCode=0 Oct 30 20:52:13.497643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.497169 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" event={"ID":"43027be5-25ba-492f-9824-3a9ae85dab5b","Type":"ContainerDied","Data":"84951baa67eedc31b8095ecf087599dfbaac7a7ffa9266905a0b43d3c9054995"} Oct 30 20:52:13.499412 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.499379 5855 generic.go:356] "Generic (PLEG): container finished" podID="621956fe-e412-4fe9-9844-bfd7993a31ca" containerID="a901405d811c6aedc4b9ccbfd7f07e7736c1768000eec91fdbc61c6ebb194db6" exitCode=0 Oct 30 20:52:13.499528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.499426 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" event={"ID":"621956fe-e412-4fe9-9844-bfd7993a31ca","Type":"ContainerDied","Data":"a901405d811c6aedc4b9ccbfd7f07e7736c1768000eec91fdbc61c6ebb194db6"} Oct 30 20:52:13.691144 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.691113 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:13.782201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.782158 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk"] Oct 30 20:52:13.782725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.782701 5855 cpu_manager.go:414] "containerMap: removing container" podUID="ae59a59e-0964-4bfa-8e6b-c498354d291a" containerName="revoke-keys" Oct 30 20:52:13.782725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.782723 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae59a59e-0964-4bfa-8e6b-c498354d291a" containerName="revoke-keys" Oct 30 20:52:13.782953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.782852 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae59a59e-0964-4bfa-8e6b-c498354d291a" containerName="revoke-keys" Oct 30 20:52:13.786860 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.786829 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:13.799612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.796645 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk"] Oct 30 20:52:13.853117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.853020 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\") pod \"ae59a59e-0964-4bfa-8e6b-c498354d291a\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " Oct 30 20:52:13.853830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.853431 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\" (UniqueName: \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\") pod \"ae59a59e-0964-4bfa-8e6b-c498354d291a\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " Oct 30 20:52:13.854277 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.854249 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\") pod \"ae59a59e-0964-4bfa-8e6b-c498354d291a\" (UID: \"ae59a59e-0964-4bfa-8e6b-c498354d291a\") " Oct 30 20:52:13.861919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.861883 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g" pod "ae59a59e-0964-4bfa-8e6b-c498354d291a" (UID: "ae59a59e-0964-4bfa-8e6b-c498354d291a"). InnerVolumeSpecName "kube-api-access-4kx2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:13.863049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.863026 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca" pod "ae59a59e-0964-4bfa-8e6b-c498354d291a" (UID: "ae59a59e-0964-4bfa-8e6b-c498354d291a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:13.865623 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.865591 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca" pod "ae59a59e-0964-4bfa-8e6b-c498354d291a" (UID: "ae59a59e-0964-4bfa-8e6b-c498354d291a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:13.955974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.955930 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:13.956181 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.955996 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:13.956181 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.956148 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwlh\" (UniqueName: \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:13.956415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.956360 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kx2g\" (UniqueName: \"kubernetes.io/projected/ae59a59e-0964-4bfa-8e6b-c498354d291a-kube-api-access-4kx2g\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:13.956415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.956391 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:13.956415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:13.956407 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae59a59e-0964-4bfa-8e6b-c498354d291a-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.057560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.057525 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.057757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.057595 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwlh\" (UniqueName: \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.057839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.057739 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.058281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.058256 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.060473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.060441 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.074570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.074544 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwlh\" (UniqueName: \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\") pod \"maas-api-revoke-sub-9f649a51-6nmvk\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.114105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.114016 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:14.442682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.442638 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk"] Oct 30 20:52:14.443776 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:14.443754 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a717127_40cf_49f8_8196_f6ab134233ba.slice/crio-efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270 WatchSource:0}: Error finding container efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270: Status 404 returned error can't find the container with id efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270 Oct 30 20:52:14.506353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.506315 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" event={"ID":"1a717127-40cf-49f8-8196-f6ab134233ba","Type":"ContainerStarted","Data":"efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270"} Oct 30 20:52:14.507841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.507819 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" Oct 30 20:52:14.507967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.507847 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s" event={"ID":"ae59a59e-0964-4bfa-8e6b-c498354d291a","Type":"ContainerDied","Data":"f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280"} Oct 30 20:52:14.507967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.507889 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5214094a152934049cce6c0325a937f4527cd929208dd7431923ee9d3719280" Oct 30 20:52:14.673893 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.673863 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:14.715400 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.715371 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:14.766911 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.766884 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\" (UniqueName: \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\") pod \"43027be5-25ba-492f-9824-3a9ae85dab5b\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " Oct 30 20:52:14.767082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.766972 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\") pod \"43027be5-25ba-492f-9824-3a9ae85dab5b\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " Oct 30 20:52:14.767082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767025 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\") pod \"43027be5-25ba-492f-9824-3a9ae85dab5b\" (UID: \"43027be5-25ba-492f-9824-3a9ae85dab5b\") " Oct 30 20:52:14.767082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767055 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\") pod \"621956fe-e412-4fe9-9844-bfd7993a31ca\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " Oct 30 20:52:14.767244 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767088 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\" (UniqueName: \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\") pod \"621956fe-e412-4fe9-9844-bfd7993a31ca\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " Oct 30 20:52:14.767244 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767126 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\") pod \"621956fe-e412-4fe9-9844-bfd7993a31ca\" (UID: \"621956fe-e412-4fe9-9844-bfd7993a31ca\") " Oct 30 20:52:14.767424 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767392 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca" pod "43027be5-25ba-492f-9824-3a9ae85dab5b" (UID: "43027be5-25ba-492f-9824-3a9ae85dab5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:14.767517 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.767471 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca" pod "621956fe-e412-4fe9-9844-bfd7993a31ca" (UID: "621956fe-e412-4fe9-9844-bfd7993a31ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:14.768175 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.768146 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.768304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.768178 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.769855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.769818 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd" pod "43027be5-25ba-492f-9824-3a9ae85dab5b" (UID: "43027be5-25ba-492f-9824-3a9ae85dab5b"). InnerVolumeSpecName "kube-api-access-t9tkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:14.769967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.769862 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca" pod "43027be5-25ba-492f-9824-3a9ae85dab5b" (UID: "43027be5-25ba-492f-9824-3a9ae85dab5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:14.769967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.769933 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca" pod "621956fe-e412-4fe9-9844-bfd7993a31ca" (UID: "621956fe-e412-4fe9-9844-bfd7993a31ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:14.769967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.769934 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq" pod "621956fe-e412-4fe9-9844-bfd7993a31ca" (UID: "621956fe-e412-4fe9-9844-bfd7993a31ca"). InnerVolumeSpecName "kube-api-access-6gwkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:14.869600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.869561 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621956fe-e412-4fe9-9844-bfd7993a31ca-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.869600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.869588 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tkd\" (UniqueName: \"kubernetes.io/projected/43027be5-25ba-492f-9824-3a9ae85dab5b-kube-api-access-t9tkd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.869600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.869600 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43027be5-25ba-492f-9824-3a9ae85dab5b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.869600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:14.869609 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gwkq\" (UniqueName: \"kubernetes.io/projected/621956fe-e412-4fe9-9844-bfd7993a31ca-kube-api-access-6gwkq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:15.515572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.515544 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" Oct 30 20:52:15.516079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.515588 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9" event={"ID":"621956fe-e412-4fe9-9844-bfd7993a31ca","Type":"ContainerDied","Data":"df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7"} Oct 30 20:52:15.516079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.515648 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df44ef621f21de9504c814d6cd0e0849177c5304128e55d681acc17860c5e2e7" Oct 30 20:52:15.517084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.517065 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" Oct 30 20:52:15.517216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.517100 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99" event={"ID":"43027be5-25ba-492f-9824-3a9ae85dab5b","Type":"ContainerDied","Data":"00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd"} Oct 30 20:52:15.517216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.517137 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00d8dead21dddaf41d0eb29cf6500c995fc9010e3390667d04634dca989c4fbd" Oct 30 20:52:15.519042 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.519020 5855 generic.go:356] "Generic (PLEG): container finished" podID="1a717127-40cf-49f8-8196-f6ab134233ba" containerID="dd0ce5ae8d43623ac4e7e3cb1531218534d9a982209893828c67f21441780643" exitCode=0 Oct 30 20:52:15.519169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:15.519068 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" event={"ID":"1a717127-40cf-49f8-8196-f6ab134233ba","Type":"ContainerDied","Data":"dd0ce5ae8d43623ac4e7e3cb1531218534d9a982209893828c67f21441780643"} Oct 30 20:52:16.674691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.674644 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:16.795204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.795173 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\") pod \"1a717127-40cf-49f8-8196-f6ab134233ba\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " Oct 30 20:52:16.795387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.795285 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\" (UniqueName: \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\") pod \"1a717127-40cf-49f8-8196-f6ab134233ba\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " Oct 30 20:52:16.795387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.795344 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\") pod \"1a717127-40cf-49f8-8196-f6ab134233ba\" (UID: \"1a717127-40cf-49f8-8196-f6ab134233ba\") " Oct 30 20:52:16.795607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.795578 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca" pod "1a717127-40cf-49f8-8196-f6ab134233ba" (UID: "1a717127-40cf-49f8-8196-f6ab134233ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:16.795793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.795772 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:16.797670 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.797622 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh" pod "1a717127-40cf-49f8-8196-f6ab134233ba" (UID: "1a717127-40cf-49f8-8196-f6ab134233ba"). InnerVolumeSpecName "kube-api-access-khwlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:16.797792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.797671 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca" pod "1a717127-40cf-49f8-8196-f6ab134233ba" (UID: "1a717127-40cf-49f8-8196-f6ab134233ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:16.896962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.896932 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khwlh\" (UniqueName: \"kubernetes.io/projected/1a717127-40cf-49f8-8196-f6ab134233ba-kube-api-access-khwlh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:16.896962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:16.896961 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a717127-40cf-49f8-8196-f6ab134233ba-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:17.532842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:17.532815 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" Oct 30 20:52:17.532842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:17.532816 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk" event={"ID":"1a717127-40cf-49f8-8196-f6ab134233ba","Type":"ContainerDied","Data":"efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270"} Oct 30 20:52:17.533097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:17.532890 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efbc38edcffdfe3c88b73fdbfcc154e9b28c8935e1b35274a9b2b2d602e7b270" Oct 30 20:52:21.703684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.703602 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq"] Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723462 5855 cpu_manager.go:414] "containerMap: removing container" podUID="621956fe-e412-4fe9-9844-bfd7993a31ca" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723499 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621956fe-e412-4fe9-9844-bfd7993a31ca" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723525 5855 cpu_manager.go:414] "containerMap: removing container" podUID="1a717127-40cf-49f8-8196-f6ab134233ba" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723534 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a717127-40cf-49f8-8196-f6ab134233ba" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723575 5855 cpu_manager.go:414] "containerMap: removing container" podUID="43027be5-25ba-492f-9824-3a9ae85dab5b" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723584 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43027be5-25ba-492f-9824-3a9ae85dab5b" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723732 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="43027be5-25ba-492f-9824-3a9ae85dab5b" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723748 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="621956fe-e412-4fe9-9844-bfd7993a31ca" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.723761 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a717127-40cf-49f8-8196-f6ab134233ba" containerName="revoke-keys" Oct 30 20:52:21.731684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.728954 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:21.747679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.744499 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:21.747679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.745273 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:52:21.747679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.745369 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:52:21.757677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.749587 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:21.757677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.752892 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq"] Oct 30 20:52:21.926682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.912763 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:21.927224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.927180 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7xp\" (UniqueName: \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:21.927470 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:21.927451 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.058679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.045936 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.059057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.059028 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7xp\" (UniqueName: \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.059215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.059201 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.060081 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.060055 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.085684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.066178 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.111388 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.111352 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7xp\" (UniqueName: \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\") pod \"maas-api-revoke-sub-3aad23c4-ldrqq\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.371557 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.371441 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:22.735336 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:22.735256 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq"] Oct 30 20:52:23.651034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:23.650993 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" event={"ID":"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45","Type":"ContainerStarted","Data":"ccc88e52898e2371c698287b024a1d5e8e9034b13ea39691529ee71c7d25ea7e"} Oct 30 20:52:23.651225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:23.651077 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" event={"ID":"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45","Type":"ContainerStarted","Data":"37bd24813c458497334dcf432b4103de50e26d65b5a12af4a65c60aff28a5d63"} Oct 30 20:52:23.672811 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:23.672729 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" podStartSLOduration=2.672709281 podStartE2EDuration="2.672709281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:23.670355675 +0000 UTC m=+757.029068422" watchObservedRunningTime="2026-10-30 20:52:23.672709281 +0000 UTC m=+757.031422031" Oct 30 20:52:24.521697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.517141 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw"] Oct 30 20:52:24.522166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.521829 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.544583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.544547 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw"] Oct 30 20:52:24.611538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.611502 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.611760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.611554 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ccq\" (UniqueName: \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.611760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.611739 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.704401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.703897 5855 generic.go:356] "Generic (PLEG): container finished" podID="b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" containerID="ccc88e52898e2371c698287b024a1d5e8e9034b13ea39691529ee71c7d25ea7e" exitCode=0 Oct 30 20:52:24.704401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.704037 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" event={"ID":"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45","Type":"ContainerDied","Data":"ccc88e52898e2371c698287b024a1d5e8e9034b13ea39691529ee71c7d25ea7e"} Oct 30 20:52:24.713923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.713242 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.713923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.713306 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ccq\" (UniqueName: \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.713923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.713353 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.723681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.714996 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.738669 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.738057 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.769131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.769064 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ccq\" (UniqueName: \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\") pod \"maas-api-revoke-sub-223d4523-ckrzw\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:24.835299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:24.835265 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:25.148131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.148102 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw"] Oct 30 20:52:25.149886 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:25.149853 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11dca7_689a_44ab_8744_91046600e2b0.slice/crio-b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729 WatchSource:0}: Error finding container b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729: Status 404 returned error can't find the container with id b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729 Oct 30 20:52:25.712705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.712581 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" event={"ID":"8b11dca7-689a-44ab-8744-91046600e2b0","Type":"ContainerStarted","Data":"fdec84f6e04ff76fbc204e73e53627812d07d9a39acedd7eacb5ef601fd59f03"} Oct 30 20:52:25.712705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.712667 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" event={"ID":"8b11dca7-689a-44ab-8744-91046600e2b0","Type":"ContainerStarted","Data":"b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729"} Oct 30 20:52:25.733436 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.733386 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" podStartSLOduration=1.7333726139999999 podStartE2EDuration="1.733372614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:25.731715766 +0000 UTC m=+759.090428513" watchObservedRunningTime="2026-10-30 20:52:25.733372614 +0000 UTC m=+759.092085360" Oct 30 20:52:25.907707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.907678 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:25.945638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.945611 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\" (UniqueName: \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\") pod \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " Oct 30 20:52:25.945865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.945729 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\") pod \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " Oct 30 20:52:25.945865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.945841 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\") pod \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\" (UID: \"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45\") " Oct 30 20:52:25.946249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.946221 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca" pod "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" (UID: "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:25.953682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.948507 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca" pod "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" (UID: "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:25.953682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:25.948557 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp" pod "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" (UID: "b067a56d-f9ed-4f57-a6d8-fd059bd6ea45"). InnerVolumeSpecName "kube-api-access-qf7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:26.047318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.047282 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.047318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.047311 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.047318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.047320 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7xp\" (UniqueName: \"kubernetes.io/projected/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45-kube-api-access-qf7xp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:26.719391 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.719363 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" Oct 30 20:52:26.719391 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.719365 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq" event={"ID":"b067a56d-f9ed-4f57-a6d8-fd059bd6ea45","Type":"ContainerDied","Data":"37bd24813c458497334dcf432b4103de50e26d65b5a12af4a65c60aff28a5d63"} Oct 30 20:52:26.719881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.719428 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bd24813c458497334dcf432b4103de50e26d65b5a12af4a65c60aff28a5d63" Oct 30 20:52:26.721177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.721151 5855 generic.go:356] "Generic (PLEG): container finished" podID="8b11dca7-689a-44ab-8744-91046600e2b0" containerID="fdec84f6e04ff76fbc204e73e53627812d07d9a39acedd7eacb5ef601fd59f03" exitCode=0 Oct 30 20:52:26.721285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.721242 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" event={"ID":"8b11dca7-689a-44ab-8744-91046600e2b0","Type":"ContainerDied","Data":"fdec84f6e04ff76fbc204e73e53627812d07d9a39acedd7eacb5ef601fd59f03"} Oct 30 20:52:26.837943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.837910 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj"] Oct 30 20:52:26.838394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.838379 5855 cpu_manager.go:414] "containerMap: removing container" podUID="b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" containerName="revoke-keys" Oct 30 20:52:26.838447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.838395 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" containerName="revoke-keys" Oct 30 20:52:26.838486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.838473 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" containerName="revoke-keys" Oct 30 20:52:26.841648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.841625 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.851427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.851398 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj"] Oct 30 20:52:26.858573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.858537 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.858848 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.858595 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.859345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.858745 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qc4\" (UniqueName: \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.961707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.960774 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qc4\" (UniqueName: \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.961707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.960925 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.961707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.960955 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.961707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.961644 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.963466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.963444 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:26.977013 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:26.976947 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qc4\" (UniqueName: \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\") pod \"maas-api-revoke-sub-5944c6c5-bppqj\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:27.160286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.160249 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:27.442671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.442620 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj"] Oct 30 20:52:27.446395 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:27.446365 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30dc773_0210_4fc6_baed_bbe38f88cc9b.slice/crio-d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2 WatchSource:0}: Error finding container d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2: Status 404 returned error can't find the container with id d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2 Oct 30 20:52:27.744117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.744006 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" event={"ID":"f30dc773-0210-4fc6-baed-bbe38f88cc9b","Type":"ContainerStarted","Data":"d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2"} Oct 30 20:52:27.893160 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.893133 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:27.972840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.972811 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\") pod \"8b11dca7-689a-44ab-8744-91046600e2b0\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " Oct 30 20:52:27.973152 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.973131 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca" pod "8b11dca7-689a-44ab-8744-91046600e2b0" (UID: "8b11dca7-689a-44ab-8744-91046600e2b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:27.973246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.973227 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\") pod \"8b11dca7-689a-44ab-8744-91046600e2b0\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " Oct 30 20:52:27.973610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.973588 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\" (UniqueName: \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\") pod \"8b11dca7-689a-44ab-8744-91046600e2b0\" (UID: \"8b11dca7-689a-44ab-8744-91046600e2b0\") " Oct 30 20:52:27.974072 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.974050 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:27.976827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.976509 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca" pod "8b11dca7-689a-44ab-8744-91046600e2b0" (UID: "8b11dca7-689a-44ab-8744-91046600e2b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:27.976936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:27.976859 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq" pod "8b11dca7-689a-44ab-8744-91046600e2b0" (UID: "8b11dca7-689a-44ab-8744-91046600e2b0"). InnerVolumeSpecName "kube-api-access-c5ccq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:28.075840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.075805 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b11dca7-689a-44ab-8744-91046600e2b0-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:28.075840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.075834 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ccq\" (UniqueName: \"kubernetes.io/projected/8b11dca7-689a-44ab-8744-91046600e2b0-kube-api-access-c5ccq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:28.751620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.751536 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" Oct 30 20:52:28.752142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.751528 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw" event={"ID":"8b11dca7-689a-44ab-8744-91046600e2b0","Type":"ContainerDied","Data":"b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729"} Oct 30 20:52:28.752142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.751711 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7abd10337d37ad45b2fe24b6030488d59eac2636aeccdd71d818c2eb1b64729" Oct 30 20:52:28.753449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.753419 5855 generic.go:356] "Generic (PLEG): container finished" podID="f30dc773-0210-4fc6-baed-bbe38f88cc9b" containerID="59771278ba55919e2495d40df156bfd395f7c3c1b53113a857117186ac73350f" exitCode=0 Oct 30 20:52:28.753602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:28.753466 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" event={"ID":"f30dc773-0210-4fc6-baed-bbe38f88cc9b","Type":"ContainerDied","Data":"59771278ba55919e2495d40df156bfd395f7c3c1b53113a857117186ac73350f"} Oct 30 20:52:29.947560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:29.947535 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:30.010455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.010337 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\") pod \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " Oct 30 20:52:30.010455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.010389 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\") pod \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " Oct 30 20:52:30.010757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.010490 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\" (UniqueName: \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\") pod \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\" (UID: \"f30dc773-0210-4fc6-baed-bbe38f88cc9b\") " Oct 30 20:52:30.010881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.010853 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca" pod "f30dc773-0210-4fc6-baed-bbe38f88cc9b" (UID: "f30dc773-0210-4fc6-baed-bbe38f88cc9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:30.011060 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.011036 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:30.012981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.012953 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca" pod "f30dc773-0210-4fc6-baed-bbe38f88cc9b" (UID: "f30dc773-0210-4fc6-baed-bbe38f88cc9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:30.013080 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.012985 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4" pod "f30dc773-0210-4fc6-baed-bbe38f88cc9b" (UID: "f30dc773-0210-4fc6-baed-bbe38f88cc9b"). InnerVolumeSpecName "kube-api-access-r5qc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:30.112221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.112183 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qc4\" (UniqueName: \"kubernetes.io/projected/f30dc773-0210-4fc6-baed-bbe38f88cc9b-kube-api-access-r5qc4\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:30.112221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.112217 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30dc773-0210-4fc6-baed-bbe38f88cc9b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:30.764992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.764953 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" event={"ID":"f30dc773-0210-4fc6-baed-bbe38f88cc9b","Type":"ContainerDied","Data":"d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2"} Oct 30 20:52:30.765169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.764997 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj" Oct 30 20:52:30.765169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:30.765046 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2afa3397a68426e7bf570451473c06f41b39f2c56b262d82aaa6de2b86298d2" Oct 30 20:52:30.926182 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:52:30.926146 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:52:31.751173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.751137 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.898632 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902461 5855 cpu_manager.go:414] "containerMap: removing container" podUID="f30dc773-0210-4fc6-baed-bbe38f88cc9b" containerName="revoke-keys" Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902491 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30dc773-0210-4fc6-baed-bbe38f88cc9b" containerName="revoke-keys" Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902529 5855 cpu_manager.go:414] "containerMap: removing container" podUID="8b11dca7-689a-44ab-8744-91046600e2b0" containerName="revoke-keys" Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902539 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b11dca7-689a-44ab-8744-91046600e2b0" containerName="revoke-keys" Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902766 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30dc773-0210-4fc6-baed-bbe38f88cc9b" containerName="revoke-keys" Oct 30 20:52:31.905679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.902786 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b11dca7-689a-44ab-8744-91046600e2b0" containerName="revoke-keys" Oct 30 20:52:31.939736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.932492 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:52:31.939736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.932642 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:31.942508 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.942469 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:52:31.964209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.964178 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:31.978539 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.978505 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:52:31.979868 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:31.979840 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-14cca2-openshift-default-dockercfg-559j6\"" Oct 30 20:52:32.009341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.009254 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5"] Oct 30 20:52:32.014219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.014192 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.026679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.019967 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:52:32.026679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.020250 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:52:32.029469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.029422 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5"] Oct 30 20:52:32.030865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.030837 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:32.030979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.030903 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:32.036453 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036426 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036484 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036531 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036556 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5vj\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036593 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.036766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036627 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.036766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036717 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036750 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.036939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036794 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.036939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036850 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fw5\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.036939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036885 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.036939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036911 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.037059 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036936 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.037059 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.036999 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.037059 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.037050 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.037147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.037100 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.037147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.037125 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.037209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.037163 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.138817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138764 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138825 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138861 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138897 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138947 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fw5\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.138980 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139003 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139030 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139077 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139106 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrx62\" (UniqueName: \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139155 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139203 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139228 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139266 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139332 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139360 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139402 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.139923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139442 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139468 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5vj\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.139923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139514 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139545 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.139923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139825 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.140170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.139979 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.140170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.140056 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.140301 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.140279 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.140740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.140718 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.141084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.141056 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.141179 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.141161 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.141239 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.141202 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.141500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.141400 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.141716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.141645 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.143717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.143672 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.144011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.143982 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.144134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.144116 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.144296 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.144280 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.152137 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.152100 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.155159 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.155123 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fw5\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.155159 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.155141 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5vj\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.156401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.156381 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.240856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.240808 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.241085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.240904 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.241085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.241064 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrx62\" (UniqueName: \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.241765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.241741 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.243785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.243762 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.254500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.254468 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrx62\" (UniqueName: \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\") pod \"maas-api-revoke-sub-f47d1e7a-gxtc5\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.282813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.276891 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:32.294683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.286524 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:32.344525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.344483 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:32.890729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.890019 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5"] Oct 30 20:52:32.899290 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:32.895277 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8bc822_203d_488f_84f0_0595ebcabe19.slice/crio-99cb61288e61eb758d84e59052cf40ce3c6f70fd3765282ecc2e19140578f1bc WatchSource:0}: Error finding container 99cb61288e61eb758d84e59052cf40ce3c6f70fd3765282ecc2e19140578f1bc: Status 404 returned error can't find the container with id 99cb61288e61eb758d84e59052cf40ce3c6f70fd3765282ecc2e19140578f1bc Oct 30 20:52:32.899290 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:32.898803 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd659143_bb47_4ba6_9842_d35c9b294c05.slice/crio-78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5 WatchSource:0}: Error finding container 78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5: Status 404 returned error can't find the container with id 78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5 Oct 30 20:52:32.899409 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.899322 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:32.899462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.899406 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:32.899462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.899454 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:32.906431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.905966 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:52:32.910622 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.910275 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:52:32.943891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.942246 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:32.943891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.942336 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:32.943891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:32.942375 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:33.835396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.834334 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" event={"ID":"dd659143-bb47-4ba6-9842-d35c9b294c05","Type":"ContainerStarted","Data":"c48980d1abad6666d5e307806aba532fef77d0636081cd8a4177d610a75d33d6"} Oct 30 20:52:33.835396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.834403 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" event={"ID":"dd659143-bb47-4ba6-9842-d35c9b294c05","Type":"ContainerStarted","Data":"78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5"} Oct 30 20:52:33.853513 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.853477 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" event={"ID":"5dc8bdd5-177f-4065-a98c-d53371cc4ab1","Type":"ContainerStarted","Data":"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac"} Oct 30 20:52:33.853870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.853827 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" event={"ID":"5dc8bdd5-177f-4065-a98c-d53371cc4ab1","Type":"ContainerStarted","Data":"d808344ac99628d28dbe281d4c27888cc9951d60e8440ba71def8fbd886f08fc"} Oct 30 20:52:33.868709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.862637 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" podStartSLOduration=2.8626165930000003 podStartE2EDuration="2.862616593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:33.861473311 +0000 UTC m=+767.220186059" watchObservedRunningTime="2026-10-30 20:52:33.862616593 +0000 UTC m=+767.221329340" Oct 30 20:52:33.943606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.943570 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" event={"ID":"2e8bc822-203d-488f-84f0-0595ebcabe19","Type":"ContainerStarted","Data":"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1"} Oct 30 20:52:33.959014 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:33.945183 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" event={"ID":"2e8bc822-203d-488f-84f0-0595ebcabe19","Type":"ContainerStarted","Data":"99cb61288e61eb758d84e59052cf40ce3c6f70fd3765282ecc2e19140578f1bc"} Oct 30 20:52:34.047845 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.040110 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podStartSLOduration=3.040087908 podStartE2EDuration="3.040087908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:33.92949966 +0000 UTC m=+767.288212430" watchObservedRunningTime="2026-10-30 20:52:34.040087908 +0000 UTC m=+767.398800655" Oct 30 20:52:34.282841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.282744 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:34.283103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.283064 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:34.283467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.283450 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:34.287828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.287804 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:34.289572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.289547 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:34.289802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.289778 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:34.744197 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.744142 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podStartSLOduration=3.7441209239999997 podStartE2EDuration="3.744120924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:34.04820306 +0000 UTC m=+767.406915795" watchObservedRunningTime="2026-10-30 20:52:34.744120924 +0000 UTC m=+768.102833671" Oct 30 20:52:34.746153 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.746121 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv"] Oct 30 20:52:34.750037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.750019 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.760103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.759607 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv"] Oct 30 20:52:34.831746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.831709 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.831746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.831749 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.831964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.831780 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcmqj\" (UniqueName: \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.932706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.932635 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.932873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.932719 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.932873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.932760 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcmqj\" (UniqueName: \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.933353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.933328 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.935311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.935289 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.947387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.947352 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcmqj\" (UniqueName: \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\") pod \"maas-api-revoke-sub-d5847cfe-smwkv\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:34.952548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.952520 5855 generic.go:356] "Generic (PLEG): container finished" podID="dd659143-bb47-4ba6-9842-d35c9b294c05" containerID="c48980d1abad6666d5e307806aba532fef77d0636081cd8a4177d610a75d33d6" exitCode=0 Oct 30 20:52:34.952739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:34.952598 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" event={"ID":"dd659143-bb47-4ba6-9842-d35c9b294c05","Type":"ContainerDied","Data":"c48980d1abad6666d5e307806aba532fef77d0636081cd8a4177d610a75d33d6"} Oct 30 20:52:35.086068 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.086018 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:35.294934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.294116 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:35.294934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.294178 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:35.295713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.295573 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:35.295713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.295622 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:35.634302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.633603 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv"] Oct 30 20:52:35.705677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.704569 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 20:52:35.761703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.749478 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 20:52:35.761703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.749609 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916511 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916590 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916700 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916772 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916804 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtp7\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916860 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.916892 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.917095 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.917716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.917153 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:35.967700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:35.966599 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" event={"ID":"98ee7f26-e162-4b4d-8bcc-e8755dd957bc","Type":"ContainerStarted","Data":"9d7d8d41788ad00d54378ece683a80131451e9dfa8e96e5933f2fabd60029693"} Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.019994 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020065 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020258 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020310 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020377 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020484 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020562 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020684 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.024680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.020716 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtp7\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.036693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.029701 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.036693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.031442 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.036693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.031740 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.036693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.034496 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.036693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.035483 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.056681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.056207 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.059204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.059181 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.062488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.062436 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtp7\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.065881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.065602 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-npbcs\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.301939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.301627 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:36.303820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.303356 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:36.303820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.303406 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:36.304201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.304112 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:36.304201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.304156 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:36.427084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.426990 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:36.552684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.551703 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\" (UniqueName: \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\") pod \"dd659143-bb47-4ba6-9842-d35c9b294c05\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " Oct 30 20:52:36.552684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.551830 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\") pod \"dd659143-bb47-4ba6-9842-d35c9b294c05\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " Oct 30 20:52:36.552684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.551882 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\") pod \"dd659143-bb47-4ba6-9842-d35c9b294c05\" (UID: \"dd659143-bb47-4ba6-9842-d35c9b294c05\") " Oct 30 20:52:36.564682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.563226 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca" pod "dd659143-bb47-4ba6-9842-d35c9b294c05" (UID: "dd659143-bb47-4ba6-9842-d35c9b294c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:36.570142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.568927 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62" pod "dd659143-bb47-4ba6-9842-d35c9b294c05" (UID: "dd659143-bb47-4ba6-9842-d35c9b294c05"). InnerVolumeSpecName "kube-api-access-nrx62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:36.570142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.569664 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca" pod "dd659143-bb47-4ba6-9842-d35c9b294c05" (UID: "dd659143-bb47-4ba6-9842-d35c9b294c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:36.666937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.654269 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrx62\" (UniqueName: \"kubernetes.io/projected/dd659143-bb47-4ba6-9842-d35c9b294c05-kube-api-access-nrx62\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:36.666937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.654304 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:36.666937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.654320 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd659143-bb47-4ba6-9842-d35c9b294c05-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:36.989809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:36.965281 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 20:52:37.002100 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.002064 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" event={"ID":"dd659143-bb47-4ba6-9842-d35c9b294c05","Type":"ContainerDied","Data":"78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5"} Oct 30 20:52:37.002300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.002128 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e1b92162e59932875c2dd0749740a5fd19f545507c79feee4f1ec75ec7c0c5" Oct 30 20:52:37.002300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.002211 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5" Oct 30 20:52:37.005533 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:37.004822 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bdc55d_6e24_4211_9244_e23b02f1d2fb.slice/crio-f88bd33ebbcb0883a95778a238962f3d7be36617d3e3cb60225ad9c17e89212f WatchSource:0}: Error finding container f88bd33ebbcb0883a95778a238962f3d7be36617d3e3cb60225ad9c17e89212f: Status 404 returned error can't find the container with id f88bd33ebbcb0883a95778a238962f3d7be36617d3e3cb60225ad9c17e89212f Oct 30 20:52:37.017692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.008534 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:37.018229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.018189 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:37.018463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.018442 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:52:37.037974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.037938 5855 generic.go:356] "Generic (PLEG): container finished" podID="98ee7f26-e162-4b4d-8bcc-e8755dd957bc" containerID="51deae9fa1a337aa2eebeaafac427b9b63cf641a401dd6cdd7aa20d3032f9edd" exitCode=0 Oct 30 20:52:37.038723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.038459 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" event={"ID":"98ee7f26-e162-4b4d-8bcc-e8755dd957bc","Type":"ContainerDied","Data":"51deae9fa1a337aa2eebeaafac427b9b63cf641a401dd6cdd7aa20d3032f9edd"} Oct 30 20:52:37.282738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.280851 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.282738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.280914 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:37.309683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.302128 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.309683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:37.302186 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:38.067504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.067468 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" event={"ID":"03bdc55d-6e24-4211-9244-e23b02f1d2fb","Type":"ContainerStarted","Data":"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d"} Oct 30 20:52:38.069337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.068230 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" event={"ID":"03bdc55d-6e24-4211-9244-e23b02f1d2fb","Type":"ContainerStarted","Data":"f88bd33ebbcb0883a95778a238962f3d7be36617d3e3cb60225ad9c17e89212f"} Oct 30 20:52:38.096967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.096335 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podStartSLOduration=3.096313124 podStartE2EDuration="3.096313124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:38.093535285 +0000 UTC m=+771.452248038" watchObservedRunningTime="2026-10-30 20:52:38.096313124 +0000 UTC m=+771.455025873" Oct 30 20:52:38.279688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.279119 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.279688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.279186 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:38.289692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.289116 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.289692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.289179 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:38.330694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.330125 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:38.330694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.330334 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.330694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.330380 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:38.795725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.762897 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:38.820679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.818283 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\") pod \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " Oct 30 20:52:38.820679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.818444 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\") pod \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " Oct 30 20:52:38.820679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.818550 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\" (UniqueName: \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\") pod \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\" (UID: \"98ee7f26-e162-4b4d-8bcc-e8755dd957bc\") " Oct 30 20:52:38.836736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.828856 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca" pod "98ee7f26-e162-4b4d-8bcc-e8755dd957bc" (UID: "98ee7f26-e162-4b4d-8bcc-e8755dd957bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:38.907465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.907418 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca" pod "98ee7f26-e162-4b4d-8bcc-e8755dd957bc" (UID: "98ee7f26-e162-4b4d-8bcc-e8755dd957bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:38.945839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.924588 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:38.946146 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.946114 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:38.946790 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:38.946757 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj" pod "98ee7f26-e162-4b4d-8bcc-e8755dd957bc" (UID: "98ee7f26-e162-4b4d-8bcc-e8755dd957bc"). InnerVolumeSpecName "kube-api-access-bcmqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:39.086682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.055823 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcmqj\" (UniqueName: \"kubernetes.io/projected/98ee7f26-e162-4b4d-8bcc-e8755dd957bc-kube-api-access-bcmqj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:39.243747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.243157 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" Oct 30 20:52:39.259682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.253888 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv" event={"ID":"98ee7f26-e162-4b4d-8bcc-e8755dd957bc","Type":"ContainerDied","Data":"9d7d8d41788ad00d54378ece683a80131451e9dfa8e96e5933f2fabd60029693"} Oct 30 20:52:39.259682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.253987 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d7d8d41788ad00d54378ece683a80131451e9dfa8e96e5933f2fabd60029693" Oct 30 20:52:39.331684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.320862 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.331684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.320933 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:39.331684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.321388 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.331684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.321423 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:39.338169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.338131 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.338480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.338442 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:39.751872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.750877 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.838758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:39.833721 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:52:40.282538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.282039 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.282538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.282098 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:40.302199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.302081 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.302199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.302136 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:40.303741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.302984 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.303741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.303028 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:40.708124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.707780 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:40.836869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.836821 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:40.837767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.837744 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:52:40.974146 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:40.973777 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:52:41.280201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.280108 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 20:52:41.280462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.280430 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 20:52:41.354053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.354015 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:41.381850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.381813 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:41.386683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.355787 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:41.386958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:41.386920 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:42.289683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.289086 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:42.289683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.289151 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 20:52:42.313689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.313100 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:42.313689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.313167 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:42.485683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.485248 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:42.489682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.486766 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:42.498695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:42.494035 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:52:43.301683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:43.300956 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:43.304714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:43.302362 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:43.304714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:43.303320 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:43.304714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:43.303365 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:43.305361 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:43.305342 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:52:44.304169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.304132 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:44.304780 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.304737 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:44.941611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.941280 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:52:44.942012 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.941991 5855 cpu_manager.go:414] "containerMap: removing container" podUID="dd659143-bb47-4ba6-9842-d35c9b294c05" containerName="revoke-keys" Oct 30 20:52:44.942274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.942015 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd659143-bb47-4ba6-9842-d35c9b294c05" containerName="revoke-keys" Oct 30 20:52:44.942274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.942040 5855 cpu_manager.go:414] "containerMap: removing container" podUID="98ee7f26-e162-4b4d-8bcc-e8755dd957bc" containerName="revoke-keys" Oct 30 20:52:44.942274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.942049 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ee7f26-e162-4b4d-8bcc-e8755dd957bc" containerName="revoke-keys" Oct 30 20:52:44.942274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.942205 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd659143-bb47-4ba6-9842-d35c9b294c05" containerName="revoke-keys" Oct 30 20:52:44.942274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.942222 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ee7f26-e162-4b4d-8bcc-e8755dd957bc" containerName="revoke-keys" Oct 30 20:52:44.948473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.948429 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:44.955126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.954873 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:52:44.955256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:44.955179 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"distinct-2-w3-cb60a2-kserve-self-signed-certs\"" Oct 30 20:52:45.079097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079057 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.079303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079123 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb799\" (UniqueName: \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.079303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079227 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.079303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079274 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.079482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079352 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.079482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.079418 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.181447 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.181532 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb799\" (UniqueName: \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.181583 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.181635 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.182208 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.182266 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.183116 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.183754 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.185682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.184018 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.196726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.196300 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.217441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.217383 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb799\" (UniqueName: \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.238682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.235256 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\") pod \"distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.286487 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.286130 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:45.304635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.304144 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:45.304635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.304201 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:45.637164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:45.637133 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:52:45.640514 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:45.640483 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093ec32c_630b_4b5b_a1a0_6cd42f66a396.slice/crio-4a73e54561c297ce40782d2928a9c31f4871fbde3f38d1fb78ad3c1cace51ba7 WatchSource:0}: Error finding container 4a73e54561c297ce40782d2928a9c31f4871fbde3f38d1fb78ad3c1cace51ba7: Status 404 returned error can't find the container with id 4a73e54561c297ce40782d2928a9c31f4871fbde3f38d1fb78ad3c1cace51ba7 Oct 30 20:52:46.276302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.276264 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz"] Oct 30 20:52:46.280559 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.280534 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.289619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.289580 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz"] Oct 30 20:52:46.291161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.291068 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:52:46.291318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.291191 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:52:46.291318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.291246 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:46.291318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.291274 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:46.304722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.303979 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:46.304722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.304031 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:52:46.409939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.408409 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6dv\" (UniqueName: \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.409939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.408506 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.409939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.408536 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.491346 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.491301 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" event={"ID":"093ec32c-630b-4b5b-a1a0-6cd42f66a396","Type":"ContainerStarted","Data":"d0b663e335ad641649fa638c1d65e29881009f479d1001566d06bbcb6ba894bd"} Oct 30 20:52:46.491502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.491361 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" event={"ID":"093ec32c-630b-4b5b-a1a0-6cd42f66a396","Type":"ContainerStarted","Data":"4a73e54561c297ce40782d2928a9c31f4871fbde3f38d1fb78ad3c1cace51ba7"} Oct 30 20:52:46.510226 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.510188 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6dv\" (UniqueName: \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.510504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.510289 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.510504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.510331 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.511341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.511311 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.514262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.514220 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podStartSLOduration=2.51420655 podStartE2EDuration="2.51420655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:46.511974749 +0000 UTC m=+779.870687501" watchObservedRunningTime="2026-10-30 20:52:46.51420655 +0000 UTC m=+779.872919337" Oct 30 20:52:46.518053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.518011 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.523940 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.523910 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6dv\" (UniqueName: \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\") pod \"maas-api-revoke-sub-aac5e068-lkvjz\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.602543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.602513 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:46.922622 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.922588 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz"] Oct 30 20:52:46.925865 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:46.925826 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dc8361_83be_4a84_addd_edc151cf0cbb.slice/crio-0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f WatchSource:0}: Error finding container 0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f: Status 404 returned error can't find the container with id 0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f Oct 30 20:52:46.994188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:46.993471 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl"] Oct 30 20:52:47.016683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.013163 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.066621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.040821 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl"] Oct 30 20:52:47.125731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.121685 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.125731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.121830 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648vk\" (UniqueName: \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.125731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.121998 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.223462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.223415 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.223720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.223517 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648vk\" (UniqueName: \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.223720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.223598 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.224279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.224239 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.226695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.226639 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.259528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.259493 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-648vk\" (UniqueName: \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\") pod \"maas-api-revoke-sub-34e8f8ef-pxnxl\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.287275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.287240 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:47.402786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.402738 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:47.510323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.510280 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" event={"ID":"e6dc8361-83be-4a84-addd-edc151cf0cbb","Type":"ContainerStarted","Data":"007a53ec3faaad22e41bb3798bfe0bfa32b9eeb714daef296c06dc93d2f2e14c"} Oct 30 20:52:47.510499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.510349 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" event={"ID":"e6dc8361-83be-4a84-addd-edc151cf0cbb","Type":"ContainerStarted","Data":"0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f"} Oct 30 20:52:47.538181 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.535007 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" podStartSLOduration=1.534988658 podStartE2EDuration="1.534988658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:47.533798013 +0000 UTC m=+780.892510760" watchObservedRunningTime="2026-10-30 20:52:47.534988658 +0000 UTC m=+780.893701408" Oct 30 20:52:47.711056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.711014 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl"] Oct 30 20:52:47.922695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.919640 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:47.922695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.921116 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:47.945029 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:47.944950 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 20:52:48.519123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:48.519086 5855 generic.go:356] "Generic (PLEG): container finished" podID="e6dc8361-83be-4a84-addd-edc151cf0cbb" containerID="007a53ec3faaad22e41bb3798bfe0bfa32b9eeb714daef296c06dc93d2f2e14c" exitCode=0 Oct 30 20:52:48.519595 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:48.519227 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" event={"ID":"e6dc8361-83be-4a84-addd-edc151cf0cbb","Type":"ContainerDied","Data":"007a53ec3faaad22e41bb3798bfe0bfa32b9eeb714daef296c06dc93d2f2e14c"} Oct 30 20:52:48.521112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:48.521079 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" event={"ID":"df46cc43-5151-48db-a5b2-f908bd89f16a","Type":"ContainerStarted","Data":"28379411682ab85a7ce0450915cda727947aa8d311dce5ca718ee05868ab9eed"} Oct 30 20:52:48.521260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:48.521141 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" event={"ID":"df46cc43-5151-48db-a5b2-f908bd89f16a","Type":"ContainerStarted","Data":"3769bc79a7e0c4d7f6a865f3213183dcbab3a87d8a854e1896f0a19b381c6777"} Oct 30 20:52:48.562355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:48.562298 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" podStartSLOduration=2.562282628 podStartE2EDuration="2.562282628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:48.55973397 +0000 UTC m=+781.918446718" watchObservedRunningTime="2026-10-30 20:52:48.562282628 +0000 UTC m=+781.920995374" Oct 30 20:52:49.553141 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.552093 5855 generic.go:356] "Generic (PLEG): container finished" podID="df46cc43-5151-48db-a5b2-f908bd89f16a" containerID="28379411682ab85a7ce0450915cda727947aa8d311dce5ca718ee05868ab9eed" exitCode=0 Oct 30 20:52:49.553141 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.552382 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" event={"ID":"df46cc43-5151-48db-a5b2-f908bd89f16a","Type":"ContainerDied","Data":"28379411682ab85a7ce0450915cda727947aa8d311dce5ca718ee05868ab9eed"} Oct 30 20:52:49.751306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.751276 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:49.862686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.862580 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\") pod \"e6dc8361-83be-4a84-addd-edc151cf0cbb\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " Oct 30 20:52:49.862843 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.862740 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\" (UniqueName: \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\") pod \"e6dc8361-83be-4a84-addd-edc151cf0cbb\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " Oct 30 20:52:49.862843 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.862822 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\") pod \"e6dc8361-83be-4a84-addd-edc151cf0cbb\" (UID: \"e6dc8361-83be-4a84-addd-edc151cf0cbb\") " Oct 30 20:52:49.863162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.863139 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca" pod "e6dc8361-83be-4a84-addd-edc151cf0cbb" (UID: "e6dc8361-83be-4a84-addd-edc151cf0cbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:49.863272 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.863252 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:49.865323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.865291 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv" pod "e6dc8361-83be-4a84-addd-edc151cf0cbb" (UID: "e6dc8361-83be-4a84-addd-edc151cf0cbb"). InnerVolumeSpecName "kube-api-access-zx6dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:49.865323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.865308 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca" pod "e6dc8361-83be-4a84-addd-edc151cf0cbb" (UID: "e6dc8361-83be-4a84-addd-edc151cf0cbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:49.964198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.964165 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6dc8361-83be-4a84-addd-edc151cf0cbb-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:49.964198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:49.964197 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6dv\" (UniqueName: \"kubernetes.io/projected/e6dc8361-83be-4a84-addd-edc151cf0cbb-kube-api-access-zx6dv\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:50.563311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.563285 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" Oct 30 20:52:50.563793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.563323 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz" event={"ID":"e6dc8361-83be-4a84-addd-edc151cf0cbb","Type":"ContainerDied","Data":"0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f"} Oct 30 20:52:50.563793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.563369 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c2a62cd8c904b770b2d45ef185e37db88e1cbbaf22dabe6b29f94bfd675257f" Oct 30 20:52:50.713511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.713485 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:50.774209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.773778 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\") pod \"df46cc43-5151-48db-a5b2-f908bd89f16a\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " Oct 30 20:52:50.774209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.773832 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\" (UniqueName: \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\") pod \"df46cc43-5151-48db-a5b2-f908bd89f16a\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " Oct 30 20:52:50.774209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.773882 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\") pod \"df46cc43-5151-48db-a5b2-f908bd89f16a\" (UID: \"df46cc43-5151-48db-a5b2-f908bd89f16a\") " Oct 30 20:52:50.774443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.774283 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca" pod "df46cc43-5151-48db-a5b2-f908bd89f16a" (UID: "df46cc43-5151-48db-a5b2-f908bd89f16a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:50.774443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.774389 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:50.777289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.777250 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk" pod "df46cc43-5151-48db-a5b2-f908bd89f16a" (UID: "df46cc43-5151-48db-a5b2-f908bd89f16a"). InnerVolumeSpecName "kube-api-access-648vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:50.777516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.777490 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca" pod "df46cc43-5151-48db-a5b2-f908bd89f16a" (UID: "df46cc43-5151-48db-a5b2-f908bd89f16a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:50.875730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.875629 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648vk\" (UniqueName: \"kubernetes.io/projected/df46cc43-5151-48db-a5b2-f908bd89f16a-kube-api-access-648vk\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:50.875730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:50.875697 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df46cc43-5151-48db-a5b2-f908bd89f16a-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:51.570309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:51.570281 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" Oct 30 20:52:51.570309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:51.570292 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl" event={"ID":"df46cc43-5151-48db-a5b2-f908bd89f16a","Type":"ContainerDied","Data":"3769bc79a7e0c4d7f6a865f3213183dcbab3a87d8a854e1896f0a19b381c6777"} Oct 30 20:52:51.570797 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:51.570344 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3769bc79a7e0c4d7f6a865f3213183dcbab3a87d8a854e1896f0a19b381c6777" Oct 30 20:52:55.291630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:55.291596 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:55.705049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:55.705018 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:55.816488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:55.816204 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:52:57.691001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.690960 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:52:57.691696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691631 5855 cpu_manager.go:414] "containerMap: removing container" podUID="df46cc43-5151-48db-a5b2-f908bd89f16a" containerName="revoke-keys" Oct 30 20:52:57.691696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691669 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df46cc43-5151-48db-a5b2-f908bd89f16a" containerName="revoke-keys" Oct 30 20:52:57.691802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691750 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e6dc8361-83be-4a84-addd-edc151cf0cbb" containerName="revoke-keys" Oct 30 20:52:57.691802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691758 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dc8361-83be-4a84-addd-edc151cf0cbb" containerName="revoke-keys" Oct 30 20:52:57.691869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691856 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="df46cc43-5151-48db-a5b2-f908bd89f16a" containerName="revoke-keys" Oct 30 20:52:57.691908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.691879 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dc8361-83be-4a84-addd-edc151cf0cbb" containerName="revoke-keys" Oct 30 20:52:57.697353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.697107 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.702641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.702598 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:52:57.719121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.719081 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"distinct-w6-3d909d-kserve-self-signed-certs\"" Oct 30 20:52:57.733342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733308 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.733498 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733385 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.733559 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733509 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.733600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733559 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.733800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733771 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.733948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.733830 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5trdr\" (UniqueName: \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.839711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.839535 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.839711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.839633 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.840051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.839977 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.840103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.840062 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5trdr\" (UniqueName: \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.840159 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.840107 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.843700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.840402 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.852720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.850412 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.852720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.851596 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.852720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.852194 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.886679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.886436 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.898217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.888064 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:57.898217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:57.893489 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5trdr\" (UniqueName: \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\") pod \"distinct-w6-3d909d-kserve-5cb87689fb-b5thf\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:58.025730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:58.023027 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:52:58.322787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:58.322741 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:52:58.326151 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:52:58.326112 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11f0200_eede_4ff2_a1aa_8bebcb373902.slice/crio-48a1beee486efffa1dd791ec2561e5987bca42c7bd2bb6b4dfa3a1b458bc06cc WatchSource:0}: Error finding container 48a1beee486efffa1dd791ec2561e5987bca42c7bd2bb6b4dfa3a1b458bc06cc: Status 404 returned error can't find the container with id 48a1beee486efffa1dd791ec2561e5987bca42c7bd2bb6b4dfa3a1b458bc06cc Oct 30 20:52:58.642312 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:58.642213 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" event={"ID":"b11f0200-eede-4ff2-a1aa-8bebcb373902","Type":"ContainerStarted","Data":"87d9fc2029a000d30371dc5e446fb75854289ef2f7811a1e3a6e20d2fd5192bd"} Oct 30 20:52:58.642312 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:58.642265 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" event={"ID":"b11f0200-eede-4ff2-a1aa-8bebcb373902","Type":"ContainerStarted","Data":"48a1beee486efffa1dd791ec2561e5987bca42c7bd2bb6b4dfa3a1b458bc06cc"} Oct 30 20:52:58.663890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:58.663832 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podStartSLOduration=1.6638152449999999 podStartE2EDuration="1.663815245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:58.661891619 +0000 UTC m=+792.020604377" watchObservedRunningTime="2026-10-30 20:52:58.663815245 +0000 UTC m=+792.022527990" Oct 30 20:52:59.027790 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:52:59.025337 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:53:00.086583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.086533 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:53:00.095242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.095202 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.101124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.101093 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:53:00.108701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.107446 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"distinct-w0-ad7ff0-kserve-self-signed-certs\"" Oct 30 20:53:00.174358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174302 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.174591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174454 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.174591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174527 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.174591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174559 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.174809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174754 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcj5j\" (UniqueName: \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.174869 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.174831 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.277569 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.277772 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.277835 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.277864 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.277977 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcj5j\" (UniqueName: \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.278044 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.278683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.278612 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.279189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.279005 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.283677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.279242 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.292635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.290479 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.296693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.296606 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcj5j\" (UniqueName: \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.322679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.320907 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\") pod \"distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.420191 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.420085 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:00.723856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:00.723762 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:53:00.728301 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:00.728265 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90e57a0c_3c51_400a_8bfc_1ab17f276b05.slice/crio-8ef09d99765a65ccae88406e9353901ae8dff1c62ae9967821c3918efbcd1080 WatchSource:0}: Error finding container 8ef09d99765a65ccae88406e9353901ae8dff1c62ae9967821c3918efbcd1080: Status 404 returned error can't find the container with id 8ef09d99765a65ccae88406e9353901ae8dff1c62ae9967821c3918efbcd1080 Oct 30 20:53:00.925264 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:53:00.925224 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:53:01.410970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.410916 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 20:53:01.415723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.415666 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.427469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.427437 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 20:53:01.498107 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498052 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498147 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncx2m\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498271 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498302 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498391 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498441 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498465 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498482 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.498715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.498505 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599678 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599736 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599761 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599788 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599822 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.599996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.599912 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600042 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncx2m\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600383 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600426 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600504 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600525 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.600948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.600924 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.601164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.601138 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.601402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.601376 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.602583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.602553 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.602836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.602820 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.613282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.613248 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.613409 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.613357 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncx2m\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.656495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.656457 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:53:01.661796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.661723 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.667639 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.667607 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" event={"ID":"90e57a0c-3c51-400a-8bfc-1ab17f276b05","Type":"ContainerStarted","Data":"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91"} Oct 30 20:53:01.667851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.667720 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" event={"ID":"90e57a0c-3c51-400a-8bfc-1ab17f276b05","Type":"ContainerStarted","Data":"8ef09d99765a65ccae88406e9353901ae8dff1c62ae9967821c3918efbcd1080"} Oct 30 20:53:01.676894 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.673643 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:53:01.702431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702396 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5lz\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702447 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702480 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702635 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702724 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702770 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702860 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.702964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.702942 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.703117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.703097 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.733835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.733770 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podStartSLOduration=1.733750535 podStartE2EDuration="1.733750535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:01.727715915 +0000 UTC m=+795.086428662" watchObservedRunningTime="2026-10-30 20:53:01.733750535 +0000 UTC m=+795.092463284" Oct 30 20:53:01.736027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.736000 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805565 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805633 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805737 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805806 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805925 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.805989 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5lz\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.806017 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.806051 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.806107 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.806531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.806488 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.808154 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.807447 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.808154 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.807741 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.809764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.809742 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.809955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.809931 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.811478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.811274 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.811478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.811452 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.822637 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.822603 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5lz\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.823079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.823055 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:01.989705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:01.989599 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:02.037194 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.037161 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 20:53:02.042019 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:02.040962 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9eae5e_4062_4950_af86_4cf931762420.slice/crio-63799ba838839a4cc4171034fe781a38adcf2ce514b37c77468dd54318a2f93d WatchSource:0}: Error finding container 63799ba838839a4cc4171034fe781a38adcf2ce514b37c77468dd54318a2f93d: Status 404 returned error can't find the container with id 63799ba838839a4cc4171034fe781a38adcf2ce514b37c77468dd54318a2f93d Oct 30 20:53:02.052365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.052220 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.052365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.052305 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.052365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.052343 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.092909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.092627 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 20:53:02.099150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.099119 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.118061 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.116708 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-3d909d-openshift-default-dockercfg-727tj\"" Oct 30 20:53:02.119081 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119050 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119175 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119123 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119178 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119278 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119230 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119279 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119324 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119382 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119355 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119388 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.119448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.119413 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5mj\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.120802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.120779 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 20:53:02.220701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.220626 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.220701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.220708 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.220979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.220741 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.220979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.220767 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5mj\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.220986 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.221054 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.221123 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.221197 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.221231 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.221931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.221892 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.222220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.222095 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.222220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.222098 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.222387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.222370 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.223097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.223062 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.224932 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.224905 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.225325 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.225298 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.238802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.238646 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.240227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.240136 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5mj\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.313973 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:02.313923 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbc8764_d0d6_498e_a3a9_6436433a249c.slice/crio-3a523f153a96e90748d5037c080868a70c2f79505370c714c64761614d91f8df WatchSource:0}: Error finding container 3a523f153a96e90748d5037c080868a70c2f79505370c714c64761614d91f8df: Status 404 returned error can't find the container with id 3a523f153a96e90748d5037c080868a70c2f79505370c714c64761614d91f8df Oct 30 20:53:02.320720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.318274 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:53:02.324037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.323995 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.324123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.324088 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.324176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.324126 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.420582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.420555 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:02.509454 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.509360 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:02.686683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.685714 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" event={"ID":"af9eae5e-4062-4950-af86-4cf931762420","Type":"ContainerStarted","Data":"c854f84492559e56279dd27563324f24bba3a5b51de30b84c7d70d9a7da85d32"} Oct 30 20:53:02.686683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.685956 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" event={"ID":"af9eae5e-4062-4950-af86-4cf931762420","Type":"ContainerStarted","Data":"63799ba838839a4cc4171034fe781a38adcf2ce514b37c77468dd54318a2f93d"} Oct 30 20:53:02.693087 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.691384 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" event={"ID":"2fbc8764-d0d6-498e-a3a9-6436433a249c","Type":"ContainerStarted","Data":"1cc6e9125e0593939b555a7493364436c7bee76443098ee60bca60c05c1382f4"} Oct 30 20:53:02.693087 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.691452 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" event={"ID":"2fbc8764-d0d6-498e-a3a9-6436433a249c","Type":"ContainerStarted","Data":"3a523f153a96e90748d5037c080868a70c2f79505370c714c64761614d91f8df"} Oct 30 20:53:02.756222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.737502 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:02.760923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.760731 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podStartSLOduration=1.760689102 podStartE2EDuration="1.760689102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:02.720003497 +0000 UTC m=+796.078716249" watchObservedRunningTime="2026-10-30 20:53:02.760689102 +0000 UTC m=+796.119416454" Oct 30 20:53:02.762015 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.761950 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podStartSLOduration=1.761930105 podStartE2EDuration="1.761930105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:02.750899508 +0000 UTC m=+796.109612261" watchObservedRunningTime="2026-10-30 20:53:02.761930105 +0000 UTC m=+796.120642857" Oct 30 20:53:02.916247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.915308 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 20:53:02.930753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.923997 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.930753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.924100 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:02.930753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:02.924140 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:03.001686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.001116 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:03.016764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.014289 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:03.016764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.014375 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:03.750685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.750312 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:03.750685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.750375 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:03.771492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.771460 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" event={"ID":"70309972-d5d4-48c2-9c39-8c76e633ad3c","Type":"ContainerStarted","Data":"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858"} Oct 30 20:53:03.771833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.771814 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" event={"ID":"70309972-d5d4-48c2-9c39-8c76e633ad3c","Type":"ContainerStarted","Data":"72eba2673e5db893ce70266d60e95062b5c65d610ee1a4603e46adb7fdf7db1e"} Oct 30 20:53:03.835680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.834629 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podStartSLOduration=1.834605831 podStartE2EDuration="1.834605831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:03.800266307 +0000 UTC m=+797.158979065" watchObservedRunningTime="2026-10-30 20:53:03.834605831 +0000 UTC m=+797.193318577" Oct 30 20:53:03.992178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.992135 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:03.992450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:03.992423 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:04.513767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.513729 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:04.514131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.514111 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:04.514297 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.514272 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:04.742838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.742801 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:04.743152 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.743121 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:04.990680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.990103 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:04.990680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:04.990168 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:05.516734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.515099 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:05.516734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.515200 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:05.744695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.736699 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:05.744695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.736777 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:05.991881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.991074 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:05.991881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:05.991180 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:06.524677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.523853 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:06.524677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.523939 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:06.740685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.740199 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:06.740685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.740277 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:06.996500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.996464 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:06.997103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:06.997070 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:07.512560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:07.512085 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:07.512560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:07.512146 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:07.785132 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:07.781791 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:07.785132 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:07.783475 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:08.035746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.031861 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.035746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.031952 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:08.055844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.055785 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:53:08.234683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.234101 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:53:08.526683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.513181 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.526683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.513265 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:08.740309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.740274 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.740613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.740581 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:53:08.997104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.997062 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.997395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:08.997365 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:53:09.139818 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.137831 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:53:09.519091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.519051 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:09.519321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.519113 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:09.972737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.966836 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:09.972737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.971264 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:09.978347 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:09.978286 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 20:53:10.229213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.229136 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:10.229903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.229877 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:10.232127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.232106 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:53:10.422538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.422481 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:10.510115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.510010 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 20:53:10.510420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.510367 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 20:53:10.512170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:10.512145 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:11.137314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:11.137281 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:53:11.516112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:11.516014 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:11.516577 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:11.516461 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:11.517244 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:11.517225 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 20:53:12.256739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:12.256676 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:53:16.971748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:16.971714 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:53:16.977364 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:16.977337 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.007515 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.007162 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:53:17.035164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035119 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035170 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035302 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035381 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgzp\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035432 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035459 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035568 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035673 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.035844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.035782 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138283 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138405 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138431 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138453 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138500 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgzp\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138534 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138551 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138618 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.138892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.138708 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.139163 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.139139 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.140001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.139347 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.140001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.139722 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.140001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.139949 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.141097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.141079 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.158638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.142208 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.161316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.161274 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.165547 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.165515 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.165799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.165767 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgzp\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.328048 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.327970 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:53:17.328476 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.328323 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:17.333827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.333785 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.362720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.360177 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452232 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452294 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbdn\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452373 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452433 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452466 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452499 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452560 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.452960 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.453791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.453115 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554210 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbdn\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554302 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554361 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554393 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554435 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554491 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554526 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554676 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.554844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554738 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.555000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554841 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.555000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.554872 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.555126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.555018 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.555126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.555044 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.555628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.555599 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.557242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.557220 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.557473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.557452 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.671214 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.671066 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbdn\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.671214 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.671066 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:17.850248 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:17.850204 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5af8da_cf9f_440d_b02a_6a473bac051a.slice/crio-3a6d43f73711c4df71124206d62177b1d4c4a818b1bde8f2bd45a9097d20b4e2 WatchSource:0}: Error finding container 3a6d43f73711c4df71124206d62177b1d4c4a818b1bde8f2bd45a9097d20b4e2: Status 404 returned error can't find the container with id 3a6d43f73711c4df71124206d62177b1d4c4a818b1bde8f2bd45a9097d20b4e2 Oct 30 20:53:17.852795 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.852766 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:53:17.853136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.853106 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:17.853182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.853166 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:17.853215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.853200 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:17.967279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:17.965746 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:18.382682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.371362 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" event={"ID":"3f5af8da-cf9f-440d-b02a-6a473bac051a","Type":"ContainerStarted","Data":"1d39054ff89bc70ae381587387ca268fbb6d0ef52d45d1937e87fc007f9eb17b"} Oct 30 20:53:18.382682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.371465 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" event={"ID":"3f5af8da-cf9f-440d-b02a-6a473bac051a","Type":"ContainerStarted","Data":"3a6d43f73711c4df71124206d62177b1d4c4a818b1bde8f2bd45a9097d20b4e2"} Oct 30 20:53:18.438702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.434764 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podStartSLOduration=2.434742783 podStartE2EDuration="2.434742783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:18.430423517 +0000 UTC m=+811.789136266" watchObservedRunningTime="2026-10-30 20:53:18.434742783 +0000 UTC m=+811.793455529" Oct 30 20:53:18.531306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.527487 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:53:18.531306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.529401 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:18.531306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.529486 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:18.531306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:18.529525 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:19.329078 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.329040 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:19.334702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.334635 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:19.334980 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.334949 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:19.398427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.398384 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" event={"ID":"37a2f120-39e0-43aa-8490-e4006c7f636c","Type":"ContainerStarted","Data":"47eb4530b2ffc818c5c276de81c0e7ea3af0733b01328f3a67ea9a31cfec1a66"} Oct 30 20:53:19.398951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.398453 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" event={"ID":"37a2f120-39e0-43aa-8490-e4006c7f636c","Type":"ContainerStarted","Data":"08aa32f5ed8677bbf38c402d200bfd3f1a003ff9d0111489a24e97d60f7b8128"} Oct 30 20:53:19.445343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.445263 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podStartSLOduration=2.445241602 podStartE2EDuration="2.445241602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:19.438146458 +0000 UTC m=+812.796859206" watchObservedRunningTime="2026-10-30 20:53:19.445241602 +0000 UTC m=+812.803954349" Oct 30 20:53:19.966678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.966381 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:19.967706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.967449 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:19.967706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:19.967510 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:20.329682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:20.329101 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:20.329682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:20.329165 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:21.010686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.010156 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.010686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.010224 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:21.337709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.331482 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.337709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.331553 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:21.966958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.966924 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.967248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:21.967221 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:22.331095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:22.331052 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:22.331840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:22.331805 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:22.969680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:22.968743 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:22.969680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:22.968811 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:23.329682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:23.329079 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:23.329682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:23.329144 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:23.967496 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:23.967052 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:23.967496 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:23.967118 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:24.332512 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:24.332441 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:24.333238 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:24.333182 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:24.966914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:24.966883 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:24.967198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:24.967162 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:25.328920 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:25.328868 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:25.329444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:25.328959 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 20:53:25.988682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:25.988268 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 20:53:25.988682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:25.988334 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 20:53:26.408681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:26.400093 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:26.408681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:26.401564 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:26.431682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:26.425296 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:53:27.283802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:27.283770 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:27.299413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:27.299385 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:27.307684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:27.304701 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:53:28.633643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.633605 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z"] Oct 30 20:53:28.638238 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.638216 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.644916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.644877 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:53:28.644916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.644878 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:53:28.644916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.644878 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:28.645198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.645183 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:53:28.647469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.647177 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z"] Oct 30 20:53:28.699343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.699306 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87s8m\" (UniqueName: \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.699538 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.699485 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.699590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.699534 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.811017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.810767 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8m\" (UniqueName: \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.811369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.811346 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.811806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.811779 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.841057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.840968 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.850211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.850181 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.855080 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.855057 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87s8m\" (UniqueName: \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\") pod \"maas-api-revoke-sub-3d11e2b1-ttx8z\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:28.956017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:28.955920 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:29.251724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:29.251635 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z"] Oct 30 20:53:29.252596 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:29.252571 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252d1bc4_5112_442d_8299_73db4dff62c2.slice/crio-8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56 WatchSource:0}: Error finding container 8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56: Status 404 returned error can't find the container with id 8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56 Oct 30 20:53:29.545210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:29.545171 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" event={"ID":"252d1bc4-5112-442d-8299-73db4dff62c2","Type":"ContainerStarted","Data":"8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56"} Oct 30 20:53:30.554751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:30.554716 5855 generic.go:356] "Generic (PLEG): container finished" podID="252d1bc4-5112-442d-8299-73db4dff62c2" containerID="9e294862d6deda734020277dcb23d8392d87c06565db21eede1c6a82d9967999" exitCode=0 Oct 30 20:53:30.555127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:30.554808 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" event={"ID":"252d1bc4-5112-442d-8299-73db4dff62c2","Type":"ContainerDied","Data":"9e294862d6deda734020277dcb23d8392d87c06565db21eede1c6a82d9967999"} Oct 30 20:53:30.929862 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:53:30.929772 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:53:32.126216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.125812 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:53:32.178823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.177329 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:53:32.178823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.177385 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:53:32.187882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.181684 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.478746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478422 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8nr\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.478746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478547 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.478746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478636 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.478746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478725 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.479077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478774 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.479077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478870 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.479077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478945 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.479077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.478978 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.479293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.479088 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.510119 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.510066 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.575595 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.575557 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:32.581062 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581022 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581232 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581104 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581232 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581144 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581232 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581218 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581288 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581322 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581423 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581521 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581502 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8nr\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.581628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.581582 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.582872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.582597 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.582995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.582950 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.583921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.583226 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.583921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.583503 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.584233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.584202 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.590255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.590226 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.598156 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.598129 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.601334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.601256 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.602397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.602010 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" event={"ID":"252d1bc4-5112-442d-8299-73db4dff62c2","Type":"ContainerDied","Data":"8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56"} Oct 30 20:53:32.602397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.602089 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8349d311d05fd0508f99eceead4f7a39802fa41ed7c110e247475c8c5b08fa56" Oct 30 20:53:32.602397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.602173 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z" Oct 30 20:53:32.608611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.604760 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:53:32.634460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.631127 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8nr\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.658700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.658313 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.683341 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\") pod \"252d1bc4-5112-442d-8299-73db4dff62c2\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.683404 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\") pod \"252d1bc4-5112-442d-8299-73db4dff62c2\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.683488 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\" (UniqueName: \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\") pod \"252d1bc4-5112-442d-8299-73db4dff62c2\" (UID: \"252d1bc4-5112-442d-8299-73db4dff62c2\") " Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.683864 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.683960 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqhf\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684028 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684058 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684131 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684245 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684294 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684347 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.684905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.684442 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.686200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.685866 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca" pod "252d1bc4-5112-442d-8299-73db4dff62c2" (UID: "252d1bc4-5112-442d-8299-73db4dff62c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:32.712613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.708641 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca" pod "252d1bc4-5112-442d-8299-73db4dff62c2" (UID: "252d1bc4-5112-442d-8299-73db4dff62c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:32.714003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.713939 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m" pod "252d1bc4-5112-442d-8299-73db4dff62c2" (UID: "252d1bc4-5112-442d-8299-73db4dff62c2"). InnerVolumeSpecName "kube-api-access-87s8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:32.787397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.785551 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.787771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.787744 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.788450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.788318 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.788699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.788682 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.789200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.789071 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.789628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.789589 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.789749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.789350 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.789932 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.789917 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqhf\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.790548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.790530 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.791050 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.791032 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.791516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.791396 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.792275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.792257 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.792484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.792456 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.793688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.793120 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:32.794251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.793827 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/252d1bc4-5112-442d-8299-73db4dff62c2-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:32.795959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.795909 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.796138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.796100 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87s8m\" (UniqueName: \"kubernetes.io/projected/252d1bc4-5112-442d-8299-73db4dff62c2-kube-api-access-87s8m\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:32.804580 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.804488 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.826486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.825363 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.827418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.827350 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqhf\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.832372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.832297 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:32.966642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:32.966603 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:33.647359 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.647162 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:53:33.803020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.802963 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:33.803256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.803232 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:33.803398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.803378 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:33.848172 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:33.848138 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab04a06c_ca83_4bb7_adf1_3135ca2f115a.slice/crio-43ac19431f30558858922536b201c0eea96738be119b7404f94e4b13a04bb22d WatchSource:0}: Error finding container 43ac19431f30558858922536b201c0eea96738be119b7404f94e4b13a04bb22d: Status 404 returned error can't find the container with id 43ac19431f30558858922536b201c0eea96738be119b7404f94e4b13a04bb22d Oct 30 20:53:33.850083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.850060 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:53:33.863187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.863149 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:33.863258 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.863238 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:33.863302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:33.863288 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:34.918683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.909891 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podStartSLOduration=2.909867861 podStartE2EDuration="2.909867861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.877230289 +0000 UTC m=+828.235943036" watchObservedRunningTime="2026-10-30 20:53:34.909867861 +0000 UTC m=+828.268580610" Oct 30 20:53:34.934390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.917888 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" event={"ID":"3d8b1f43-ed6a-4134-a242-bff6301be868","Type":"ContainerStarted","Data":"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1"} Oct 30 20:53:34.955540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.952941 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" event={"ID":"3d8b1f43-ed6a-4134-a242-bff6301be868","Type":"ContainerStarted","Data":"e26f45993c4902188ca712c2aeb07843d894aeaf2a0b10ae30cc94b7b0b6e0d7"} Oct 30 20:53:34.955540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.953020 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" event={"ID":"ab04a06c-ca83-4bb7-adf1-3135ca2f115a","Type":"ContainerStarted","Data":"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d"} Oct 30 20:53:34.955540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.953060 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" event={"ID":"ab04a06c-ca83-4bb7-adf1-3135ca2f115a","Type":"ContainerStarted","Data":"43ac19431f30558858922536b201c0eea96738be119b7404f94e4b13a04bb22d"} Oct 30 20:53:34.981969 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.981931 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:34.982348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.982325 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:34.982529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:34.982506 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:35.660642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:35.660602 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:35.662057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:35.662030 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:35.662197 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:35.662090 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:35.998742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:35.990156 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:35.998742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:35.990233 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:36.661936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:36.661898 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:36.662255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:36.662178 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:36.969128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:36.969047 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:36.969267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:36.969125 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:37.660465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:37.660123 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:37.660465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:37.660199 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:37.969690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:37.969561 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:37.969690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:37.969637 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:38.660038 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:38.660003 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:38.660275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:38.660070 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:38.977757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:38.972437 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:38.977757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:38.972512 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:39.667680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:39.661144 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:39.667680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:39.661233 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:39.968748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:39.968358 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:39.968748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:39.968405 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:40.662745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:40.660737 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:40.662745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:40.660808 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:40.968124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:40.968044 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:40.968266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:40.968121 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:41.660920 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:41.660878 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:41.661097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:41.660957 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:53:41.968554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:41.968460 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:53:41.968554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:41.968523 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:53:42.862675 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.862619 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:42.863873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.863849 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:42.866807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.866782 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:53:42.898968 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.898901 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podStartSLOduration=10.89888082 podStartE2EDuration="10.89888082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.92862801 +0000 UTC m=+828.287340755" watchObservedRunningTime="2026-10-30 20:53:42.89888082 +0000 UTC m=+836.257593558" Oct 30 20:53:42.974255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.974225 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:42.975528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.975506 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:42.976330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:42.976312 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:53:46.536727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.536648 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 20:53:46.537741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.537701 5855 cpu_manager.go:414] "containerMap: removing container" podUID="252d1bc4-5112-442d-8299-73db4dff62c2" containerName="revoke-keys" Oct 30 20:53:46.537741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.537726 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252d1bc4-5112-442d-8299-73db4dff62c2" containerName="revoke-keys" Oct 30 20:53:46.537919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.537892 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="252d1bc4-5112-442d-8299-73db4dff62c2" containerName="revoke-keys" Oct 30 20:53:46.545907 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.545873 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 20:53:46.546058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.546014 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.574887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.574842 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmq9\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.574969 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575062 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575090 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575184 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575315 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575234 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575308 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575425 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575407 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.575481 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.575463 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676322 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676389 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676434 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676479 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676517 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676616 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676725 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676778 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmq9\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676840 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.676958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.676885 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.677208 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.677178 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.677379 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.677232 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.677693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.677637 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.677801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.677726 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.680084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.680056 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.680299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.680284 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.699178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.699145 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.699375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.699350 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmq9\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:46.865476 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:46.865434 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:47.170592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.170560 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 20:53:47.171272 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:47.171249 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076643e8_713c_4c2f_acd3_651855e837c3.slice/crio-8acd3ea2439b2bf6cdb26531a2b34d0ea9c5b52f8a857a1dddcefaeee00ac444 WatchSource:0}: Error finding container 8acd3ea2439b2bf6cdb26531a2b34d0ea9c5b52f8a857a1dddcefaeee00ac444: Status 404 returned error can't find the container with id 8acd3ea2439b2bf6cdb26531a2b34d0ea9c5b52f8a857a1dddcefaeee00ac444 Oct 30 20:53:47.178083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.178036 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.178263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.178140 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.178263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.178198 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.221896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.221392 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:53:47.232609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.229959 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.245304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.237329 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:53:47.297172 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.297108 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303029 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303001 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303244 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwr4\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303392 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303551 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303681 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.303913 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.303891 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.304150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.304131 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.305605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.305037 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.410628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.410586 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.411554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.411510 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.411951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.411919 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.412781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.412763 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.412958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.412941 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwr4\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.413092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.413079 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.413276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.413263 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.413405 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.413391 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.413601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.413583 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.414387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.414371 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.414523 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.412461 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.415434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.415416 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.415565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.412711 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.416418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.416385 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.430764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.425159 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.430764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.427468 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.430764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.430079 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwr4\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.430764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.430089 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.623629 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.623581 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:47.928857 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:47.928826 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e4c813_9f4b_4049_b86c_d7beefe32ca5.slice/crio-67411d76f67adef2f0bd7ed4a4788e35a59bc1568c25a91722712dc2bce587fc WatchSource:0}: Error finding container 67411d76f67adef2f0bd7ed4a4788e35a59bc1568c25a91722712dc2bce587fc: Status 404 returned error can't find the container with id 67411d76f67adef2f0bd7ed4a4788e35a59bc1568c25a91722712dc2bce587fc Oct 30 20:53:47.929733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.929677 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:53:47.931769 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.931733 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.931880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.931809 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:47.931880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:47.931859 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:53:48.148131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.148092 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" event={"ID":"076643e8-713c-4c2f-acd3-651855e837c3","Type":"ContainerStarted","Data":"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0"} Oct 30 20:53:48.148342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.148165 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" event={"ID":"076643e8-713c-4c2f-acd3-651855e837c3","Type":"ContainerStarted","Data":"8acd3ea2439b2bf6cdb26531a2b34d0ea9c5b52f8a857a1dddcefaeee00ac444"} Oct 30 20:53:48.151804 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.151524 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" event={"ID":"79e4c813-9f4b-4049-b86c-d7beefe32ca5","Type":"ContainerStarted","Data":"67411d76f67adef2f0bd7ed4a4788e35a59bc1568c25a91722712dc2bce587fc"} Oct 30 20:53:48.178121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.178039 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podStartSLOduration=2.178014753 podStartE2EDuration="2.178014753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:48.17279707 +0000 UTC m=+841.531509810" watchObservedRunningTime="2026-10-30 20:53:48.178014753 +0000 UTC m=+841.536727544" Oct 30 20:53:48.642932 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.642890 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:53:48.652488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.652454 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:53:48.652741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.652591 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.665491 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.665426 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"distinct-2-w0-ad7ff0-kserve-self-signed-certs\"" Oct 30 20:53:48.751475 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751435 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.751736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751485 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.751736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751607 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.751736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751724 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvwj\" (UniqueName: \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.751938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751778 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.751938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.751835 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860411 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvwj\" (UniqueName: \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860471 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860538 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860685 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860721 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.860887 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.861408 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.862681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.862070 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.892682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.882391 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.892682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.882771 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.916841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.902810 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:48.916841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.902892 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:48.917447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.917388 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:48.921739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.920230 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:48.924400 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:48.924323 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvwj\" (UniqueName: \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\") pod \"distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:49.037112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.037072 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd"] Oct 30 20:53:49.056537 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.056340 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:49.082430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.069504 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.144429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.143326 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd"] Oct 30 20:53:49.152903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.150500 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:53:49.152903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.150623 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:53:49.152903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.150696 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:53:49.152903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.150764 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:53:49.316285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.313932 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wv5\" (UniqueName: \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.316285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.314093 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.316285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.314130 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.337682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.337468 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" event={"ID":"79e4c813-9f4b-4049-b86c-d7beefe32ca5","Type":"ContainerStarted","Data":"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3"} Oct 30 20:53:49.371565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.371499 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podStartSLOduration=2.371475849 podStartE2EDuration="2.371475849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:49.370412945 +0000 UTC m=+842.729125692" watchObservedRunningTime="2026-10-30 20:53:49.371475849 +0000 UTC m=+842.730188596" Oct 30 20:53:49.424525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.424487 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.424767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.424747 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.425139 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.425117 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wv5\" (UniqueName: \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.426812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.426787 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.465671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.465613 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.465976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.465953 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wv5\" (UniqueName: \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\") pod \"maas-api-revoke-sub-2aa2339f-82tdd\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.560759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.560723 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:49.631699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.624227 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:49.640682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.633557 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:49.640682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.633636 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:49.644801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.644769 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:53:49.647347 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:49.647316 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c739b88_a880_4edb_81ab_3deb0ff4ad8e.slice/crio-e924b7f60ae960bbd4f8aa3e822423a277542c8205735f39c03bda1b79a342fa WatchSource:0}: Error finding container e924b7f60ae960bbd4f8aa3e822423a277542c8205735f39c03bda1b79a342fa: Status 404 returned error can't find the container with id e924b7f60ae960bbd4f8aa3e822423a277542c8205735f39c03bda1b79a342fa Oct 30 20:53:49.907472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.892880 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:49.907472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:49.892940 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:50.344690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.340292 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd"] Oct 30 20:53:50.352631 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.352273 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" event={"ID":"0c739b88-a880-4edb-81ab-3deb0ff4ad8e","Type":"ContainerStarted","Data":"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546"} Oct 30 20:53:50.352631 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.352324 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" event={"ID":"0c739b88-a880-4edb-81ab-3deb0ff4ad8e","Type":"ContainerStarted","Data":"e924b7f60ae960bbd4f8aa3e822423a277542c8205735f39c03bda1b79a342fa"} Oct 30 20:53:50.395757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.395080 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podStartSLOduration=2.395055702 podStartE2EDuration="2.395055702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:50.372304833 +0000 UTC m=+843.731017576" watchObservedRunningTime="2026-10-30 20:53:50.395055702 +0000 UTC m=+843.753768450" Oct 30 20:53:50.627684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.627110 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:50.627684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.627174 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:50.894682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.875719 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:50.894682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:50.875779 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:51.057553 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.057516 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:51.366692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.365035 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" event={"ID":"2c515b39-3fd8-4acf-9317-1689fc97d88f","Type":"ContainerStarted","Data":"b1aeda06b3e534dbc0c7cbacbffa7f88c4000796df2176e22d0ab2bf8133ec51"} Oct 30 20:53:51.366692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.365097 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" event={"ID":"2c515b39-3fd8-4acf-9317-1689fc97d88f","Type":"ContainerStarted","Data":"401821e01419abf49e588ae31a950ebf721302cc67f5e579954b661c4ee77851"} Oct 30 20:53:51.406729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.399094 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" podStartSLOduration=2.399069967 podStartE2EDuration="2.399069967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:51.393082578 +0000 UTC m=+844.751795328" watchObservedRunningTime="2026-10-30 20:53:51.399069967 +0000 UTC m=+844.757782714" Oct 30 20:53:51.624691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.624172 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:51.624691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.624238 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:51.870263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.870231 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:51.870463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:51.870288 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:52.435683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.433496 5855 generic.go:356] "Generic (PLEG): container finished" podID="2c515b39-3fd8-4acf-9317-1689fc97d88f" containerID="b1aeda06b3e534dbc0c7cbacbffa7f88c4000796df2176e22d0ab2bf8133ec51" exitCode=0 Oct 30 20:53:52.435683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.434595 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" event={"ID":"2c515b39-3fd8-4acf-9317-1689fc97d88f","Type":"ContainerDied","Data":"b1aeda06b3e534dbc0c7cbacbffa7f88c4000796df2176e22d0ab2bf8133ec51"} Oct 30 20:53:52.633691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.629574 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:52.633691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.629694 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:52.872372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.870922 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:52.872372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.870973 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:52.936693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.933710 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc"] Oct 30 20:53:52.945687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.941976 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:52.960214 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:52.958143 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc"] Oct 30 20:53:53.042570 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.042509 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9ps\" (UniqueName: \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.047324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.043027 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.049698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.047604 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.153779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.153158 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9ps\" (UniqueName: \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.153779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.153331 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.153779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.153356 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.156717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.155154 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.158218 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.158191 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.167189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.167158 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9ps\" (UniqueName: \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\") pod \"maas-api-revoke-sub-cd40a544-kp9tc\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.332677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.332435 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:53.626682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.626084 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:53.626682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.626145 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:53.934701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.873274 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:53.934701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.873333 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:53.975744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:53.971597 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc"] Oct 30 20:53:54.451934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.447950 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 20:53:54.458707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.458177 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.458994 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.458970 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:54.465920 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.465868 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"distinct-2-w4-f2ed1e-kserve-self-signed-certs\"" Oct 30 20:53:54.468571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.468250 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 20:53:54.513681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.511232 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" event={"ID":"0f76e035-c571-41e6-a7a1-c2a5f251ef4b","Type":"ContainerStarted","Data":"a45b422cb2d2a25f42c33781e1862c32730effcc75ddbd633fbe93a27fff9afd"} Oct 30 20:53:54.513681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.513309 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" event={"ID":"2c515b39-3fd8-4acf-9317-1689fc97d88f","Type":"ContainerDied","Data":"401821e01419abf49e588ae31a950ebf721302cc67f5e579954b661c4ee77851"} Oct 30 20:53:54.513681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.513365 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401821e01419abf49e588ae31a950ebf721302cc67f5e579954b661c4ee77851" Oct 30 20:53:54.513681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.513440 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd" Oct 30 20:53:54.631684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.627984 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:54.631684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.628046 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665511 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\") pod \"2c515b39-3fd8-4acf-9317-1689fc97d88f\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665562 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\") pod \"2c515b39-3fd8-4acf-9317-1689fc97d88f\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665596 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\" (UniqueName: \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\") pod \"2c515b39-3fd8-4acf-9317-1689fc97d88f\" (UID: \"2c515b39-3fd8-4acf-9317-1689fc97d88f\") " Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665921 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665946 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.665977 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpqd\" (UniqueName: \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.666080 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.666172 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.677682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.666222 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.698472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.679319 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca" pod "2c515b39-3fd8-4acf-9317-1689fc97d88f" (UID: "2c515b39-3fd8-4acf-9317-1689fc97d88f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:54.698472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.679879 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca" pod "2c515b39-3fd8-4acf-9317-1689fc97d88f" (UID: "2c515b39-3fd8-4acf-9317-1689fc97d88f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:54.698472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.694778 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5" pod "2c515b39-3fd8-4acf-9317-1689fc97d88f" (UID: "2c515b39-3fd8-4acf-9317-1689fc97d88f"). InnerVolumeSpecName "kube-api-access-x5wv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:54.780682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.779705 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.780682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.779901 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.780682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.780324 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.780682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.780354 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.780682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.780481 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpqd\" (UniqueName: \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.781130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.780998 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.790821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.781872 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:54.790821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.781896 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c515b39-3fd8-4acf-9317-1689fc97d88f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:54.790821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.781911 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wv5\" (UniqueName: \"kubernetes.io/projected/2c515b39-3fd8-4acf-9317-1689fc97d88f-kube-api-access-x5wv5\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:54.790821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.782559 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.790821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.782984 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.799781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.793559 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.799781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.793916 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.803677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.802575 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:54.893595 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.893511 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:54.893890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.893858 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:54.953461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:54.953395 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpqd\" (UniqueName: \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\") pod \"distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:55.099090 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.099050 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:55.544083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.544041 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" event={"ID":"0f76e035-c571-41e6-a7a1-c2a5f251ef4b","Type":"ContainerStarted","Data":"6abff57efdf5a5030128f690e24779c45156d9d963a9bf6cd5c221729c7ea8b7"} Oct 30 20:53:55.566912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.565897 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" podStartSLOduration=3.565880361 podStartE2EDuration="3.565880361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:55.564717082 +0000 UTC m=+848.923429828" watchObservedRunningTime="2026-10-30 20:53:55.565880361 +0000 UTC m=+848.924593108" Oct 30 20:53:55.633576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.630766 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:55.633576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.630828 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:55.871681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.871536 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:53:55.871681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.871596 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:53:55.904966 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:55.898488 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 20:53:56.568486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.567969 5855 generic.go:356] "Generic (PLEG): container finished" podID="0f76e035-c571-41e6-a7a1-c2a5f251ef4b" containerID="6abff57efdf5a5030128f690e24779c45156d9d963a9bf6cd5c221729c7ea8b7" exitCode=0 Oct 30 20:53:56.568486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.568163 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" event={"ID":"0f76e035-c571-41e6-a7a1-c2a5f251ef4b","Type":"ContainerDied","Data":"6abff57efdf5a5030128f690e24779c45156d9d963a9bf6cd5c221729c7ea8b7"} Oct 30 20:53:56.572516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.571699 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" event={"ID":"904d6f64-339e-4a4c-b265-9bb2ac9a63a1","Type":"ContainerStarted","Data":"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a"} Oct 30 20:53:56.572516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.571749 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" event={"ID":"904d6f64-339e-4a4c-b265-9bb2ac9a63a1","Type":"ContainerStarted","Data":"eb9cd6f66d414cd09e5615ec29c1627a613e19aaf51fe5efe8f627d1a9d53ca9"} Oct 30 20:53:56.624031 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.623984 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:53:56.624300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.624272 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 20:53:56.628713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.628541 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podStartSLOduration=2.628521278 podStartE2EDuration="2.628521278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:56.625189683 +0000 UTC m=+849.983902453" watchObservedRunningTime="2026-10-30 20:53:56.628521278 +0000 UTC m=+849.987234025" Oct 30 20:53:56.888174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.888015 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:56.889441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.889419 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:56.890765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.890715 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 20:53:56.893927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:56.893905 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:53:57.100268 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:57.100230 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:53:57.630150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:57.630115 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:57.630878 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:57.630853 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:57.631695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:57.631645 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:53:58.053740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.053515 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6"] Oct 30 20:53:58.061825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.054464 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2c515b39-3fd8-4acf-9317-1689fc97d88f" containerName="revoke-keys" Oct 30 20:53:58.061825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.054493 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c515b39-3fd8-4acf-9317-1689fc97d88f" containerName="revoke-keys" Oct 30 20:53:58.061825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.054632 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c515b39-3fd8-4acf-9317-1689fc97d88f" containerName="revoke-keys" Oct 30 20:53:58.093841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.093519 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.119722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.101370 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6"] Oct 30 20:53:58.225685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.199895 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:58.293431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.267862 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.293431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.267987 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.293431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.268079 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z47v\" (UniqueName: \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.386756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.373325 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\") pod \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " Oct 30 20:53:58.435423 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.435382 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\") pod \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " Oct 30 20:53:58.436197 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.436174 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\" (UniqueName: \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\") pod \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\" (UID: \"0f76e035-c571-41e6-a7a1-c2a5f251ef4b\") " Oct 30 20:53:58.437109 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.437084 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.437918 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.437896 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.438894 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.438875 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z47v\" (UniqueName: \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.439420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.430919 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca" pod "0f76e035-c571-41e6-a7a1-c2a5f251ef4b" (UID: "0f76e035-c571-41e6-a7a1-c2a5f251ef4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:58.439546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.438761 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.440013 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.439994 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.531960 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.509744 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.532284 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.532251 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z47v\" (UniqueName: \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\") pod \"maas-api-revoke-sub-d107461b-z48l6\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.532464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.532402 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps" pod "0f76e035-c571-41e6-a7a1-c2a5f251ef4b" (UID: "0f76e035-c571-41e6-a7a1-c2a5f251ef4b"). InnerVolumeSpecName "kube-api-access-sz9ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:58.532964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.532703 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca" pod "0f76e035-c571-41e6-a7a1-c2a5f251ef4b" (UID: "0f76e035-c571-41e6-a7a1-c2a5f251ef4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:53:58.542762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.542558 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9ps\" (UniqueName: \"kubernetes.io/projected/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-kube-api-access-sz9ps\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.542762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.542629 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f76e035-c571-41e6-a7a1-c2a5f251ef4b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:58.580698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.578509 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:53:58.630877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.626324 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" Oct 30 20:53:58.636304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.636267 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc" event={"ID":"0f76e035-c571-41e6-a7a1-c2a5f251ef4b","Type":"ContainerDied","Data":"a45b422cb2d2a25f42c33781e1862c32730effcc75ddbd633fbe93a27fff9afd"} Oct 30 20:53:58.636587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:58.636567 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a45b422cb2d2a25f42c33781e1862c32730effcc75ddbd633fbe93a27fff9afd" Oct 30 20:53:59.064153 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.064086 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:59.725844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.722749 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:53:59.785401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.741929 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:59.785401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.772406 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:59.816876 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:53:59.816821 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b87802_7223_4d66_b819_ce8f0ee5279d.slice/crio-fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58 WatchSource:0}: Error finding container fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58: Status 404 returned error can't find the container with id fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58 Oct 30 20:53:59.846701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.821794 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6"] Oct 30 20:53:59.869684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.866820 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:59.905251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.904595 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:53:59.972017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:53:59.971985 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:00.598883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.598828 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:00.765881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.765265 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.779009 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.778951 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.847682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.843983 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" event={"ID":"25b87802-7223-4d66-b819-ce8f0ee5279d","Type":"ContainerStarted","Data":"b39a3e171fbacfdea14f07a11e4b7899b702bbc101266b87fdcffadf15190833"} Oct 30 20:54:00.847682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.844054 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" event={"ID":"25b87802-7223-4d66-b819-ce8f0ee5279d","Type":"ContainerStarted","Data":"fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58"} Oct 30 20:54:00.857831 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.857789 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.874006 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.873937 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" podStartSLOduration=2.873916656 podStartE2EDuration="2.873916656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:00.871105525 +0000 UTC m=+854.229818298" watchObservedRunningTime="2026-10-30 20:54:00.873916656 +0000 UTC m=+854.232629407" Oct 30 20:54:00.971802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:00.969806 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:01.017429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.016808 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.150750 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:54:01.144438 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:54:01.198034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.174873 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.226382 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.226340 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:01.449865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.447097 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.627684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.603783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.627684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.604432 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:01.663964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.663928 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 20:54:01.691885 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.687733 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:01.746877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.746810 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:01.951020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.785741 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.954381 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.851767 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.954822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.888432 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.888467 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.954998 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.301485 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" event={"ID":"25b87802-7223-4d66-b819-ce8f0ee5279d","Type":"ContainerDied","Data":"b39a3e171fbacfdea14f07a11e4b7899b702bbc101266b87fdcffadf15190833"} Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.301577 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.301621 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.301677 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.048380 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.048449 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.061816 5855 generic.go:356] "Generic (PLEG): container finished" podID="25b87802-7223-4d66-b819-ce8f0ee5279d" containerID="b39a3e171fbacfdea14f07a11e4b7899b702bbc101266b87fdcffadf15190833" exitCode=0 Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:01.955259 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.318733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.185631 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.337833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.321947 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:02.382211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.382184 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:02.382441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.382430 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:02.400487 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.229006 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:02.402427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.402398 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:02.801989 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.801945 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:02.994687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.970937 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.994687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.971204 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:54:02.994687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:02.971798 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:03.141952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.141822 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded" Oct 30 20:54:03.305600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.305556 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:03.305990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.305972 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.314339 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.314591 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.315151 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.315439 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.317982 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:03.354687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.323881 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:54:03.406741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.403167 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:03.406741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.406090 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:54:03.406741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.406145 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:03.509864 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.509814 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:03.822274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.822242 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:03.845431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.844971 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:03.913178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.913131 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:03.937836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.914149 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:03.974393 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:03.974350 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:54:04.323679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.320783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.323679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.321376 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:04.323679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.321587 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:04.323679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.321827 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:54:04.323679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.321976 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:04.338696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.336451 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.450981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.450287 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:54:04.475643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.475523 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.764392 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.764290 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.852953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.851800 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.985199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.943567 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:04.985199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.943976 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:04.988899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:04.988858 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:05.027120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.017740 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:05.095110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.092798 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8"] Oct 30 20:54:05.101223 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.101191 5855 cpu_manager.go:414] "containerMap: removing container" podUID="0f76e035-c571-41e6-a7a1-c2a5f251ef4b" containerName="revoke-keys" Oct 30 20:54:05.238703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.237914 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f76e035-c571-41e6-a7a1-c2a5f251ef4b" containerName="revoke-keys" Oct 30 20:54:05.238703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.238191 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f76e035-c571-41e6-a7a1-c2a5f251ef4b" containerName="revoke-keys" Oct 30 20:54:05.248027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.122873 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:54:05.291695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.291601 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:05.291994 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.291979 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8"] Oct 30 20:54:05.292147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.292136 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:05.292246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.292237 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:05.294210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.291940 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.311520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.309254 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:05.357683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.357569 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.381562 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\") pod \"25b87802-7223-4d66-b819-ce8f0ee5279d\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.381837 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\" (UniqueName: \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\") pod \"25b87802-7223-4d66-b819-ce8f0ee5279d\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.381876 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\") pod \"25b87802-7223-4d66-b819-ce8f0ee5279d\" (UID: \"25b87802-7223-4d66-b819-ce8f0ee5279d\") " Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.382328 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.382368 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.382873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.382409 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdwm\" (UniqueName: \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.394156 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.394107 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca" pod "25b87802-7223-4d66-b819-ce8f0ee5279d" (UID: "25b87802-7223-4d66-b819-ce8f0ee5279d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:05.541543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.541504 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:05.549683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.545889 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.549991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.549959 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.550185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.550167 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdwm\" (UniqueName: \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.554697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.550647 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:05.586747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.579258 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.588053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.588025 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v" pod "25b87802-7223-4d66-b819-ce8f0ee5279d" (UID: "25b87802-7223-4d66-b819-ce8f0ee5279d"). InnerVolumeSpecName "kube-api-access-9z47v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:05.680701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.665082 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.680701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.669458 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdwm\" (UniqueName: \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\") pod \"maas-api-revoke-sub-5e4b13ce-cmqp8\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:05.732349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.732305 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z47v\" (UniqueName: \"kubernetes.io/projected/25b87802-7223-4d66-b819-ce8f0ee5279d-kube-api-access-9z47v\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:05.761855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.761805 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca" pod "25b87802-7223-4d66-b819-ce8f0ee5279d" (UID: "25b87802-7223-4d66-b819-ce8f0ee5279d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:05.796728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.792229 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:05.803184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.802932 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:05.815007 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.814979 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" Oct 30 20:54:05.824716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.816325 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6" event={"ID":"25b87802-7223-4d66-b819-ce8f0ee5279d","Type":"ContainerDied","Data":"fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58"} Oct 30 20:54:05.824716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.816380 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4a128ab4d58baa8d5f5f632f90656c7b65b5465815718691c990971124bd58" Oct 30 20:54:05.843563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.842141 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b87802-7223-4d66-b819-ce8f0ee5279d-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:05.854686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.854522 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:05.922268 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:05.922233 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:06.229943 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:06.229867 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f06a27_247f_4f12_b59d_4ce04aa60ae5.slice/crio-6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd WatchSource:0}: Error finding container 6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd: Status 404 returned error can't find the container with id 6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd Oct 30 20:54:06.230434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:06.230389 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8"] Oct 30 20:54:06.823422 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:06.823388 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" event={"ID":"d3f06a27-247f-4f12-b59d-4ce04aa60ae5","Type":"ContainerStarted","Data":"42df6366cdd4abcdfef1deface41cd21bf104bb257af3e780d1478095bdced32"} Oct 30 20:54:06.823729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:06.823453 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" event={"ID":"d3f06a27-247f-4f12-b59d-4ce04aa60ae5","Type":"ContainerStarted","Data":"6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd"} Oct 30 20:54:06.845536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:06.845442 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" podStartSLOduration=2.8454274760000002 podStartE2EDuration="2.845427476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:06.841700235 +0000 UTC m=+860.200412984" watchObservedRunningTime="2026-10-30 20:54:06.845427476 +0000 UTC m=+860.204140221" Oct 30 20:54:07.848157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:07.848110 5855 generic.go:356] "Generic (PLEG): container finished" podID="d3f06a27-247f-4f12-b59d-4ce04aa60ae5" containerID="42df6366cdd4abcdfef1deface41cd21bf104bb257af3e780d1478095bdced32" exitCode=0 Oct 30 20:54:07.848687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:07.848205 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" event={"ID":"d3f06a27-247f-4f12-b59d-4ce04aa60ae5","Type":"ContainerDied","Data":"42df6366cdd4abcdfef1deface41cd21bf104bb257af3e780d1478095bdced32"} Oct 30 20:54:09.173600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.173565 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:09.239807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.239758 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\" (UniqueName: \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\") pod \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " Oct 30 20:54:09.239983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.239832 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\") pod \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " Oct 30 20:54:09.239983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.239970 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\") pod \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\" (UID: \"d3f06a27-247f-4f12-b59d-4ce04aa60ae5\") " Oct 30 20:54:09.240365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.240333 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca" pod "d3f06a27-247f-4f12-b59d-4ce04aa60ae5" (UID: "d3f06a27-247f-4f12-b59d-4ce04aa60ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:09.240771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.240745 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:09.242202 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.242180 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca" pod "d3f06a27-247f-4f12-b59d-4ce04aa60ae5" (UID: "d3f06a27-247f-4f12-b59d-4ce04aa60ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:09.242307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.242282 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm" pod "d3f06a27-247f-4f12-b59d-4ce04aa60ae5" (UID: "d3f06a27-247f-4f12-b59d-4ce04aa60ae5"). InnerVolumeSpecName "kube-api-access-frdwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:09.341949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.341914 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdwm\" (UniqueName: \"kubernetes.io/projected/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-kube-api-access-frdwm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:09.341949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.341946 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f06a27-247f-4f12-b59d-4ce04aa60ae5-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:09.864872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.864838 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" event={"ID":"d3f06a27-247f-4f12-b59d-4ce04aa60ae5","Type":"ContainerDied","Data":"6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd"} Oct 30 20:54:09.865023 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.864887 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ac53fd93fb566ae9a954d5534fccc2db0650347ef344bb7ff8196499c226cbd" Oct 30 20:54:09.865023 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:09.864853 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8" Oct 30 20:54:12.880442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.880398 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn"] Oct 30 20:54:12.881150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881128 5855 cpu_manager.go:414] "containerMap: removing container" podUID="25b87802-7223-4d66-b819-ce8f0ee5279d" containerName="revoke-keys" Oct 30 20:54:12.881246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881153 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b87802-7223-4d66-b819-ce8f0ee5279d" containerName="revoke-keys" Oct 30 20:54:12.881246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881227 5855 cpu_manager.go:414] "containerMap: removing container" podUID="d3f06a27-247f-4f12-b59d-4ce04aa60ae5" containerName="revoke-keys" Oct 30 20:54:12.881246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881236 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f06a27-247f-4f12-b59d-4ce04aa60ae5" containerName="revoke-keys" Oct 30 20:54:12.881390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881365 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b87802-7223-4d66-b819-ce8f0ee5279d" containerName="revoke-keys" Oct 30 20:54:12.881390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.881387 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f06a27-247f-4f12-b59d-4ce04aa60ae5" containerName="revoke-keys" Oct 30 20:54:12.885228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.885205 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:12.893688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.889582 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:12.893688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.892034 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:54:12.893688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.892143 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:54:12.893688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.892226 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:54:12.896703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:12.894930 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn"] Oct 30 20:54:13.012432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.012381 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnbd\" (UniqueName: \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.012628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.012584 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.012717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.012638 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.114349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.114303 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.114528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.114366 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.114587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.114573 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnbd\" (UniqueName: \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.115088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.115065 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.117055 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.117031 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.127799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.127777 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnbd\" (UniqueName: \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\") pod \"maas-api-revoke-sub-cde9a306-9snkn\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.225671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.225510 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:13.661047 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:13.661013 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cd4639_18bd_49b2_9e59_f4562f63df27.slice/crio-66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b WatchSource:0}: Error finding container 66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b: Status 404 returned error can't find the container with id 66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b Oct 30 20:54:13.662781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.662712 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn"] Oct 30 20:54:13.921897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:13.921806 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" event={"ID":"89cd4639-18bd-49b2-9e59-f4562f63df27","Type":"ContainerStarted","Data":"66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b"} Oct 30 20:54:14.931223 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:14.931143 5855 generic.go:356] "Generic (PLEG): container finished" podID="89cd4639-18bd-49b2-9e59-f4562f63df27" containerID="e911993e35734c489eb298a900a3a10b5510590f50efa108179dd84e84decee3" exitCode=0 Oct 30 20:54:14.931600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:14.931237 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" event={"ID":"89cd4639-18bd-49b2-9e59-f4562f63df27","Type":"ContainerDied","Data":"e911993e35734c489eb298a900a3a10b5510590f50efa108179dd84e84decee3"} Oct 30 20:54:16.178792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.178150 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:16.299473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.296080 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\") pod \"89cd4639-18bd-49b2-9e59-f4562f63df27\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " Oct 30 20:54:16.299473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.296215 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\") pod \"89cd4639-18bd-49b2-9e59-f4562f63df27\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " Oct 30 20:54:16.299473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.296265 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\" (UniqueName: \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\") pod \"89cd4639-18bd-49b2-9e59-f4562f63df27\" (UID: \"89cd4639-18bd-49b2-9e59-f4562f63df27\") " Oct 30 20:54:16.301519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.301446 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca" pod "89cd4639-18bd-49b2-9e59-f4562f63df27" (UID: "89cd4639-18bd-49b2-9e59-f4562f63df27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:16.302244 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.302217 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd" pod "89cd4639-18bd-49b2-9e59-f4562f63df27" (UID: "89cd4639-18bd-49b2-9e59-f4562f63df27"). InnerVolumeSpecName "kube-api-access-6bnbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:16.306200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.306168 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca" pod "89cd4639-18bd-49b2-9e59-f4562f63df27" (UID: "89cd4639-18bd-49b2-9e59-f4562f63df27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:16.398698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.398605 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:16.398698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.398643 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnbd\" (UniqueName: \"kubernetes.io/projected/89cd4639-18bd-49b2-9e59-f4562f63df27-kube-api-access-6bnbd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:16.398698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.398668 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cd4639-18bd-49b2-9e59-f4562f63df27-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:16.698905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.698805 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72"] Oct 30 20:54:16.699419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.699401 5855 cpu_manager.go:414] "containerMap: removing container" podUID="89cd4639-18bd-49b2-9e59-f4562f63df27" containerName="revoke-keys" Oct 30 20:54:16.699419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.699417 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cd4639-18bd-49b2-9e59-f4562f63df27" containerName="revoke-keys" Oct 30 20:54:16.699580 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.699511 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cd4639-18bd-49b2-9e59-f4562f63df27" containerName="revoke-keys" Oct 30 20:54:16.703390 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.703358 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.711755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.711724 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72"] Oct 30 20:54:16.805916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.805876 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kbq\" (UniqueName: \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.806060 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.805995 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.806100 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.806059 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.909895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.908808 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kbq\" (UniqueName: \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.909895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.908916 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.909895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.908965 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.919681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.915770 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.923679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.920628 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.937682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.934562 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kbq\" (UniqueName: \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\") pod \"maas-api-revoke-sub-fef9f8d2-p7h72\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:16.979746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.976596 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" Oct 30 20:54:16.979746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.976629 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn" event={"ID":"89cd4639-18bd-49b2-9e59-f4562f63df27","Type":"ContainerDied","Data":"66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b"} Oct 30 20:54:16.979746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:16.976697 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b83440b4b3e7e7870df97d5090fe9930aa4cce87b47f9ca12bc305a770799b" Oct 30 20:54:17.024176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.024136 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:17.327177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.327144 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72"] Oct 30 20:54:17.331449 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:17.331417 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f2862c_3758_4aa1_8bd4_ab3d2ee25e4f.slice/crio-6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c WatchSource:0}: Error finding container 6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c: Status 404 returned error can't find the container with id 6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c Oct 30 20:54:17.339130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.339107 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 20:54:17.347722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.346279 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.365877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.365185 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417424 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417516 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417601 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417668 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417695 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417813 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417907 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4f7\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.417966 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.420043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.418073 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519405 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519523 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519577 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519634 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519708 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519764 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519809 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4f7\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.519959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519853 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.520201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.519964 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.521771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.520413 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.521771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.520519 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.521771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.520823 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.521771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.521450 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.522098 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.521815 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.575265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.573904 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.575265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.574225 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.575265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.574228 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.575265 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.575156 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4f7\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:17.723130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:17.723092 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:18.062703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.062312 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" event={"ID":"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f","Type":"ContainerStarted","Data":"ec2f445e1df274eaae1f9b164869d92703141728f8fcb005e4f45a0937984297"} Oct 30 20:54:18.062703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.062389 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" event={"ID":"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f","Type":"ContainerStarted","Data":"6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c"} Oct 30 20:54:18.115590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.114996 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 20:54:18.148829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.148695 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" podStartSLOduration=2.14867124 podStartE2EDuration="2.14867124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:18.09768307 +0000 UTC m=+871.456395843" watchObservedRunningTime="2026-10-30 20:54:18.14867124 +0000 UTC m=+871.507383980" Oct 30 20:54:18.183933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.179755 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:18.183933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.179858 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:18.183933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:18.179906 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:19.099682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.099393 5855 generic.go:356] "Generic (PLEG): container finished" podID="c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" containerID="ec2f445e1df274eaae1f9b164869d92703141728f8fcb005e4f45a0937984297" exitCode=0 Oct 30 20:54:19.099682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.099513 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" event={"ID":"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f","Type":"ContainerDied","Data":"ec2f445e1df274eaae1f9b164869d92703141728f8fcb005e4f45a0937984297"} Oct 30 20:54:19.113683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.110426 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" event={"ID":"e15834ab-9d48-49b1-b6be-98655a5ec37c","Type":"ContainerStarted","Data":"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07"} Oct 30 20:54:19.113683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.110505 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" event={"ID":"e15834ab-9d48-49b1-b6be-98655a5ec37c","Type":"ContainerStarted","Data":"2faa7678b3a9922701d01675d361f165df641a644eb0e09f216e6bef6803879e"} Oct 30 20:54:19.173685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.172974 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podStartSLOduration=2.172954571 podStartE2EDuration="2.172954571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:19.170577518 +0000 UTC m=+872.529290264" watchObservedRunningTime="2026-10-30 20:54:19.172954571 +0000 UTC m=+872.531667322" Oct 30 20:54:19.309715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.290463 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.309715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.300274 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:19.341712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.328519 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.369798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.368939 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.474695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.472349 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:19.732787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.732260 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:19.732787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.732392 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:19.733182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.732988 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:19.836841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.836783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.837534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.837093 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:19.867704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.866999 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.932703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.922937 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.999085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:19.998977 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.069852 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.069801 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.302906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.302862 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.304923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.304898 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:20.328834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.328784 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.329193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.329172 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.329608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.329585 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.330887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.330866 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:20.373215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.373174 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.373839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.373814 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:20.471682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.465994 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:20.542709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.538782 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.550715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.547158 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:54:20.784743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.777038 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:54:20.877293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.818837 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:20.877293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.818912 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:20.935875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.919604 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:20.935875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.920120 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:54:20.948200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.942581 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:20.948200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:20.943738 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.091783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.082377 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.331012 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.330970 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.418738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.415546 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:21.447693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.445067 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.480834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.449807 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:21.480834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.450899 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:21.480834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.455170 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.714742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.673794 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.719679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.715059 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.723680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.722431 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 20:54:21.789765 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.496066 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.792591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.792558 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:21.792879 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.792864 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:21.793035 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.793023 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:21.793206 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.793177 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:21.818685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.799108 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:21.867485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.867458 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:21.867823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.867803 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:21.867993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.867980 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:21.895371 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.820281 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:21.960300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.960260 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:21.960543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.920836 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:21.989890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.957336 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.989890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.920755 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.989890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:21.957363 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:22.121684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.102440 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.435700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.423013 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.477868 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.477814 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.478534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.478504 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:22.501477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.485278 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:22.501477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.485309 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:22.501477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.485423 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:22.715815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.715695 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.767014 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.762833 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.798710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.795694 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:22.850966 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.850921 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.890575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.887180 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:22.890575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.887261 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:23.007429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.993783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.007429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.994243 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:54:23.007429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:22.994547 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:54:23.072996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.050711 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:23.109947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.105630 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.116284 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.116238 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.521673 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.465426 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:23.521673 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.510589 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:23.541676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.536817 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:23.627685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.614137 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:23.745877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.745835 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:23.746480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.623271 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:23.755362 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.755326 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca" pod "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" (UID: "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:23.755541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.652857 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:23.757456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.703295 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\") pod \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " Oct 30 20:54:23.768027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.767980 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\") pod \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " Oct 30 20:54:23.768416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.768389 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\" (UniqueName: \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\") pod \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\" (UID: \"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f\") " Oct 30 20:54:23.769120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.769088 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.769504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.623217 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Oct 30 20:54:23.770510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.770486 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca" pod "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" (UID: "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:23.785494 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.770727 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:23.842708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.834063 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:23.842708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.770793 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:23.842708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.623293 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:23.869449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.869414 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:23.999122 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:23.999054 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.050745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.026291 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:24.140058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.140010 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.166729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.060553 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.208984 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.167091 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.311396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.307583 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" event={"ID":"c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f","Type":"ContainerDied","Data":"6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c"} Oct 30 20:54:24.340523 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.340466 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bae11b72861918b32d6c44ce4e7767e7e56ad4973a7df3a3ab0c834a0534d6c" Oct 30 20:54:24.342637 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.342608 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72" Oct 30 20:54:24.482478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.482417 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:24.539897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.539827 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.645683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.620009 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq" pod "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" (UID: "c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f"). InnerVolumeSpecName "kube-api-access-84kbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:24.752729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.673780 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kbq\" (UniqueName: \"kubernetes.io/projected/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f-kube-api-access-84kbq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:24.752999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.734777 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:24.753173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.753143 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:24.753286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.738848 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.769337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.767126 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.769337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.767474 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:24.770921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.770878 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.771195 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.771169 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.841681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.836876 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.849735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.849033 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:24.875590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.875561 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:24.938717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:24.924282 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:25.035688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.023991 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.174749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.165286 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:25.221748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.167748 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.497441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.496770 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.547702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.544594 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:25.547702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.544677 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:25.547702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.546028 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:25.547702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.546556 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:54:25.551002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.550981 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:25.565823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.565797 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:25.589794 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.589760 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:25.723642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.723605 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:25.723910 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:25.723690 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:26.723793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:26.723762 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:26.724189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:26.723811 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:27.724348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:27.724310 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:27.724874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:27.724372 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:28.525698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.525635 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv"] Oct 30 20:54:28.526499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.526479 5855 cpu_manager.go:414] "containerMap: removing container" podUID="c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" containerName="revoke-keys" Oct 30 20:54:28.526564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.526502 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" containerName="revoke-keys" Oct 30 20:54:28.526650 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.526638 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" containerName="revoke-keys" Oct 30 20:54:28.532908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.532880 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.536995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.536965 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv"] Oct 30 20:54:28.537856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.537835 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:54:28.539095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.539065 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:54:28.539095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.539090 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:54:28.539262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.539073 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:28.710776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.710735 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs596\" (UniqueName: \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.710979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.710956 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.711116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.711096 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.723525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.723484 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:28.723802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.723537 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:28.813681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.813266 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.813681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.813383 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.813681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.813516 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs596\" (UniqueName: \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.814761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.814728 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.817510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.817474 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.826921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.826887 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs596\" (UniqueName: \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\") pod \"maas-api-revoke-sub-4eee999e-fkfnv\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:28.853229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:28.853193 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:29.343626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.343589 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz"] Oct 30 20:54:29.347836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.347815 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.357558 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.356417 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz"] Oct 30 20:54:29.410995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.410023 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv"] Oct 30 20:54:29.413806 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:29.413764 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606505b8_108e_47ea_8833_d5145896da78.slice/crio-1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918 WatchSource:0}: Error finding container 1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918: Status 404 returned error can't find the container with id 1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918 Oct 30 20:54:29.437202 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.437160 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.437520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.437497 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.437599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.437583 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz549\" (UniqueName: \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.538882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.538846 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.539085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.539064 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.539166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.539142 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz549\" (UniqueName: \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.540026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.539991 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.596681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.586974 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.733699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.723935 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:29.733699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.724012 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:29.784917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.784886 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz549\" (UniqueName: \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\") pod \"maas-api-revoke-sub-e127d293-lsjzz\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:29.804406 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.804366 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" event={"ID":"606505b8-108e-47ea-8833-d5145896da78","Type":"ContainerStarted","Data":"1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918"} Oct 30 20:54:29.967216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:29.967126 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:30.296701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.296640 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz"] Oct 30 20:54:30.724927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.724882 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:54:30.725088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.724944 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 20:54:30.819337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.819249 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" event={"ID":"99de9906-c352-4212-b527-3f215c951208","Type":"ContainerStarted","Data":"ff03e21397a2ad72ba984757f1451c859e468b9d25b7cac2ae673e1be2c90b92"} Oct 30 20:54:30.819337 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.819318 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" event={"ID":"99de9906-c352-4212-b527-3f215c951208","Type":"ContainerStarted","Data":"b7c7a339cb29b48ae8a7d4b5e8a9f56bc94274d3e9b90b5852e94f570bc4f36f"} Oct 30 20:54:30.821563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.821531 5855 generic.go:356] "Generic (PLEG): container finished" podID="606505b8-108e-47ea-8833-d5145896da78" containerID="fdb0fed42bb0848bc61c09264cd4fa8eb91f7dd76c0634145241e13e5bed1a72" exitCode=0 Oct 30 20:54:30.821722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.821595 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" event={"ID":"606505b8-108e-47ea-8833-d5145896da78","Type":"ContainerDied","Data":"fdb0fed42bb0848bc61c09264cd4fa8eb91f7dd76c0634145241e13e5bed1a72"} Oct 30 20:54:30.841801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:30.841736 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" podStartSLOduration=1.841715437 podStartE2EDuration="1.841715437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:30.838297941 +0000 UTC m=+884.197010687" watchObservedRunningTime="2026-10-30 20:54:30.841715437 +0000 UTC m=+884.200428184" Oct 30 20:54:30.928272 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:54:30.928237 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:54:31.799953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:31.799909 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:31.800748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:31.800724 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:31.803178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:31.803154 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 20:54:31.836896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:31.836859 5855 generic.go:356] "Generic (PLEG): container finished" podID="99de9906-c352-4212-b527-3f215c951208" containerID="ff03e21397a2ad72ba984757f1451c859e468b9d25b7cac2ae673e1be2c90b92" exitCode=0 Oct 30 20:54:31.837069 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:31.836951 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" event={"ID":"99de9906-c352-4212-b527-3f215c951208","Type":"ContainerDied","Data":"ff03e21397a2ad72ba984757f1451c859e468b9d25b7cac2ae673e1be2c90b92"} Oct 30 20:54:32.169184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.169157 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:32.257353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.257320 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\" (UniqueName: \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\") pod \"606505b8-108e-47ea-8833-d5145896da78\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " Oct 30 20:54:32.257532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.257396 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\") pod \"606505b8-108e-47ea-8833-d5145896da78\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " Oct 30 20:54:32.257532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.257515 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\") pod \"606505b8-108e-47ea-8833-d5145896da78\" (UID: \"606505b8-108e-47ea-8833-d5145896da78\") " Oct 30 20:54:32.257806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.257783 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca" pod "606505b8-108e-47ea-8833-d5145896da78" (UID: "606505b8-108e-47ea-8833-d5145896da78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:32.258084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.258056 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/606505b8-108e-47ea-8833-d5145896da78-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:32.260454 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.260429 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca" pod "606505b8-108e-47ea-8833-d5145896da78" (UID: "606505b8-108e-47ea-8833-d5145896da78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:32.260566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.260459 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596" pod "606505b8-108e-47ea-8833-d5145896da78" (UID: "606505b8-108e-47ea-8833-d5145896da78"). InnerVolumeSpecName "kube-api-access-cs596". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:32.347307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.347267 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 20:54:32.347867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.347853 5855 cpu_manager.go:414] "containerMap: removing container" podUID="606505b8-108e-47ea-8833-d5145896da78" containerName="revoke-keys" Oct 30 20:54:32.347934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.347868 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606505b8-108e-47ea-8833-d5145896da78" containerName="revoke-keys" Oct 30 20:54:32.347996 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.347986 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="606505b8-108e-47ea-8833-d5145896da78" containerName="revoke-keys" Oct 30 20:54:32.351837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.351816 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.359357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.359325 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs596\" (UniqueName: \"kubernetes.io/projected/606505b8-108e-47ea-8833-d5145896da78-kube-api-access-cs596\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:32.359509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.359362 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/606505b8-108e-47ea-8833-d5145896da78-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:32.372424 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.372370 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 20:54:32.460810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.460716 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmz8\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.460810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.460762 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.460848 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461037 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461131 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461202 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461259 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461327 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461310 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.461384 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.461344 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.524162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.524125 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:54:32.528751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.528726 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.540011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.539978 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:54:32.545498 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.545465 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:54:32.550617 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.550587 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.564249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564212 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564294 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564408 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564437 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564479 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564602 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmz8\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564635 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564693 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.564963 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.564835 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.565263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.565244 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.587679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.570678 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:54:32.587679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.572080 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.587679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.572296 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.607745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.589100 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.607745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.589813 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.607745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.598275 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.624016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.611698 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.624016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.619264 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.624016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.622287 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmz8\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.666417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666374 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666423 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.666635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666456 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666541 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vj9\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666637 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666681 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666734 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.666853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666772 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.666853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666829 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666877 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.666904 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.667088 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667059 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.667220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667169 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667202 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.667310 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667236 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:32.667310 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667284 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667315 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqvp\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667347 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.667426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.667399 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774075 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774136 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqvp\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774174 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774201 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774293 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774319 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774353 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774380 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vj9\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774408 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774431 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774466 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774523 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774559 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774610 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774649 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774710 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774866 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.774920 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.815206 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.816312 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.816589 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.817407 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.817946 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.818171 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.819002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.818421 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.826720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.819617 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.826720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.819802 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.826720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.819928 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.826720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.820689 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.826720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.825107 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.832102 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.830537 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.844812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.839241 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.844812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.841697 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.844812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.844107 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.844812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.844248 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqvp\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.845169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.844909 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vj9\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:32.881686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.881299 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" Oct 30 20:54:32.885031 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.884683 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv" event={"ID":"606505b8-108e-47ea-8833-d5145896da78","Type":"ContainerDied","Data":"1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918"} Oct 30 20:54:32.885031 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.884748 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ada2725085a21822fb715bf23b6575afb65c817b2e6c835bc568ea5502bd918" Oct 30 20:54:32.926116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.925886 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:32.931373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:32.931248 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:33.083840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.075101 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:33.099050 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.098939 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 20:54:33.106541 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:33.106497 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0bf314_5df2_425e_9bee_131a00148753.slice/crio-a0bf08e9bc321bd5e358321e5b6e0ade5e5348bfebcd4839c45e95fdd417dba3 WatchSource:0}: Error finding container a0bf08e9bc321bd5e358321e5b6e0ade5e5348bfebcd4839c45e95fdd417dba3: Status 404 returned error can't find the container with id a0bf08e9bc321bd5e358321e5b6e0ade5e5348bfebcd4839c45e95fdd417dba3 Oct 30 20:54:33.110264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.110220 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.110358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.110315 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.110394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.110357 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.183791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.183755 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\") pod \"99de9906-c352-4212-b527-3f215c951208\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " Oct 30 20:54:33.184006 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.183990 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\") pod \"99de9906-c352-4212-b527-3f215c951208\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " Oct 30 20:54:33.184090 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.184078 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\" (UniqueName: \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\") pod \"99de9906-c352-4212-b527-3f215c951208\" (UID: \"99de9906-c352-4212-b527-3f215c951208\") " Oct 30 20:54:33.184432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.184403 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca" pod "99de9906-c352-4212-b527-3f215c951208" (UID: "99de9906-c352-4212-b527-3f215c951208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:33.184835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.184816 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99de9906-c352-4212-b527-3f215c951208-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.186237 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.186206 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca" pod "99de9906-c352-4212-b527-3f215c951208" (UID: "99de9906-c352-4212-b527-3f215c951208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:33.186351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.186337 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549" pod "99de9906-c352-4212-b527-3f215c951208" (UID: "99de9906-c352-4212-b527-3f215c951208"). InnerVolumeSpecName "kube-api-access-gz549". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:33.278332 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.277567 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:54:33.283967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.283934 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:54:33.294331 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:33.294254 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec75fe6_1787_4b66_96a5_7c2412ae7414.slice/crio-9b5d1db2b714f556c16d851697657c32a720c97fbdb7de6eda4bb94b007f7277 WatchSource:0}: Error finding container 9b5d1db2b714f556c16d851697657c32a720c97fbdb7de6eda4bb94b007f7277: Status 404 returned error can't find the container with id 9b5d1db2b714f556c16d851697657c32a720c97fbdb7de6eda4bb94b007f7277 Oct 30 20:54:33.297600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.297538 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz549\" (UniqueName: \"kubernetes.io/projected/99de9906-c352-4212-b527-3f215c951208-kube-api-access-gz549\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.297600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.297571 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99de9906-c352-4212-b527-3f215c951208-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.302970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.301582 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.302970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.301695 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.302970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.301745 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.308851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.308122 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.308851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.308210 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.308851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.308253 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:33.896680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.895808 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" event={"ID":"fe64f471-65f4-49d9-b086-5d98363146a4","Type":"ContainerStarted","Data":"7fed4ef894fdbcad8bfff29c119a73c2ec9580320fb11db41129a6303c4eeb6a"} Oct 30 20:54:33.896680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.895885 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" event={"ID":"fe64f471-65f4-49d9-b086-5d98363146a4","Type":"ContainerStarted","Data":"2f7c8ef83c3dc21cebe4f24cc2637822e8cd794e7e2636b74a467ef779f4e92d"} Oct 30 20:54:33.898292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.898234 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" event={"ID":"cec75fe6-1787-4b66-96a5-7c2412ae7414","Type":"ContainerStarted","Data":"e98a81def1e70644b7092547b360db36e70969672302d2b2ca894f21a819cd44"} Oct 30 20:54:33.898454 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.898307 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" event={"ID":"cec75fe6-1787-4b66-96a5-7c2412ae7414","Type":"ContainerStarted","Data":"9b5d1db2b714f556c16d851697657c32a720c97fbdb7de6eda4bb94b007f7277"} Oct 30 20:54:33.901606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.901258 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" event={"ID":"1b0bf314-5df2-425e-9bee-131a00148753","Type":"ContainerStarted","Data":"7f3f861b0924688e616fd3eb92163baf75baae4a692371ddbce5763681374f12"} Oct 30 20:54:33.901606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.901327 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" event={"ID":"1b0bf314-5df2-425e-9bee-131a00148753","Type":"ContainerStarted","Data":"a0bf08e9bc321bd5e358321e5b6e0ade5e5348bfebcd4839c45e95fdd417dba3"} Oct 30 20:54:33.903616 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.903589 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" event={"ID":"99de9906-c352-4212-b527-3f215c951208","Type":"ContainerDied","Data":"b7c7a339cb29b48ae8a7d4b5e8a9f56bc94274d3e9b90b5852e94f570bc4f36f"} Oct 30 20:54:33.903770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.903635 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c7a339cb29b48ae8a7d4b5e8a9f56bc94274d3e9b90b5852e94f570bc4f36f" Oct 30 20:54:33.903770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.903639 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz" Oct 30 20:54:33.926804 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.926358 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:33.927566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.927504 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podStartSLOduration=1.927483286 podStartE2EDuration="1.927483286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:33.923726378 +0000 UTC m=+887.282439105" watchObservedRunningTime="2026-10-30 20:54:33.927483286 +0000 UTC m=+887.286196037" Oct 30 20:54:33.931748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.931696 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:33.988434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:33.988369 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podStartSLOduration=1.988346417 podStartE2EDuration="1.988346417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:33.974466543 +0000 UTC m=+887.333179289" watchObservedRunningTime="2026-10-30 20:54:33.988346417 +0000 UTC m=+887.347059165" Oct 30 20:54:34.020686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.007077 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podStartSLOduration=2.007044889 podStartE2EDuration="2.007044889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:34.005132024 +0000 UTC m=+887.363844772" watchObservedRunningTime="2026-10-30 20:54:34.007044889 +0000 UTC m=+887.365757686" Oct 30 20:54:34.691680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.691315 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:34.732696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.700113 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:34.732696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.700174 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:34.943750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.942798 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:34.943750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.942861 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:34.953687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.951856 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:34.953687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:34.951938 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:35.676680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.669502 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:35.676680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.669582 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:35.935621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.935529 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:35.935905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.935873 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:35.940011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.939982 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:35.940236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:35.940207 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:36.008937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.008784 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:36.672683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.671761 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:36.672683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.671820 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:36.771676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.763993 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:36.986360 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.986265 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:36.986635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.986599 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:36.989887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.989842 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:36.990151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:36.990123 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:37.669739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.668858 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:37.669739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.668936 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:37.929767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.926296 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:37.929767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.926361 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:37.937682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.936872 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:37.937682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:37.936937 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:38.473683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.472814 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:38.665509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.665300 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:38.677401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.668123 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:38.678290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.678245 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:38.766142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.766039 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:38.926911 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.926874 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:38.927188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.927156 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:38.955683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.932985 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:38.955683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:38.933054 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:39.001682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.000747 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.475683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.474760 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.475683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.475245 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:39.554681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.553767 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.579733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.565510 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:39.693683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.689900 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.693683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.690791 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:39.751756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.742702 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:39.751756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.742765 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:39.765171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.765104 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.805695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.792024 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.805695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.792426 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:39.874684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.842243 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:39.982707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.972088 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:39.982707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.972150 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:39.982707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.973328 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:39.982707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:39.973373 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:40.019800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.009173 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:54:40.019800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.017862 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:40.056803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.056230 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.505812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.505763 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.697037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.690043 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:40.697037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.690129 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:40.697037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.690570 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.776609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.772796 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:54:40.800964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.800917 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.874681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.828786 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.874681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.844500 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:40.879240 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.868784 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:40.942293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.942251 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:40.942498 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.942315 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:40.985523 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.970945 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:40.985523 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.971006 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:40.985523 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.975884 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:40.986842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.986811 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:40.987386 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:40.987366 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:41.030763 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.015056 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:41.063747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.063689 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:41.066117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.066087 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:41.401245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.401215 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:41.484718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.482239 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:41.550848 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.550818 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:41.557478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.557448 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:41.560418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.560372 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:41.582516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.582487 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:41.668582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.668075 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:41.668582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.668135 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:41.747149 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.746162 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7"] Oct 30 20:54:41.762123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.760909 5855 cpu_manager.go:414] "containerMap: removing container" podUID="99de9906-c352-4212-b527-3f215c951208" containerName="revoke-keys" Oct 30 20:54:41.762123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.760935 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99de9906-c352-4212-b527-3f215c951208" containerName="revoke-keys" Oct 30 20:54:41.762123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.761142 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="99de9906-c352-4212-b527-3f215c951208" containerName="revoke-keys" Oct 30 20:54:41.769709 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.769679 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7"] Oct 30 20:54:41.769883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.769836 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:41.789483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.789235 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:54:41.789483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.789365 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:54:41.789483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.789419 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:41.807776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.791504 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:54:41.976732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.974414 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:41.976732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.974570 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:41.976732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:41.974630 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sft75\" (UniqueName: \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.080502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.078738 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 20:54:42.080502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.078857 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 20:54:42.083731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.082305 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:54:42.083731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.082427 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 20:54:42.149229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.149180 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.149450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.149378 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.149450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.149414 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sft75\" (UniqueName: \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.150671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.150622 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.153678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.153627 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.163641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.163603 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sft75\" (UniqueName: \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\") pod \"maas-api-revoke-sub-a20e0148-dfzz7\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.425556 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.425181 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:42.682744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.676470 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 20:54:42.682744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.676531 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 20:54:43.008719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:42.999300 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7"] Oct 30 20:54:43.156696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.156190 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt"] Oct 30 20:54:43.187700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.183221 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.314757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.312922 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.314757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.313054 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65w4f\" (UniqueName: \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.314757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.313147 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.365702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.349065 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt"] Oct 30 20:54:43.474711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.425207 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.474711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.425329 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65w4f\" (UniqueName: \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.474711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.425406 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.487779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.475422 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.500686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.500569 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.504344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.504274 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65w4f\" (UniqueName: \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\") pod \"maas-api-revoke-sub-d5421489-f8qlt\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.510092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.509953 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" event={"ID":"412de240-9d58-4923-8926-6001b1d33f7d","Type":"ContainerStarted","Data":"fd0bc796d57040d04d4cf5b6bf6c511862fe1282b4db77f5dbe3bc2dd1f71fed"} Oct 30 20:54:43.541324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.536881 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:43.800641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.798921 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:43.800641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.800549 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:43.809495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.808843 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:54:43.823343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.823314 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6"] Oct 30 20:54:43.836324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.836290 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:43.922749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.901038 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6"] Oct 30 20:54:43.977682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.976381 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:43.997687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:43.980437 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:44.019247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.015634 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.019247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.015736 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.019247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.015779 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4f5\" (UniqueName: \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.101136 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.115449 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.118076 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.122013 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.122079 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.122128 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4f5\" (UniqueName: \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.127686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.124861 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.128247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.127900 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 20:54:44.158124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.153350 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.158124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.153422 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt"] Oct 30 20:54:44.158124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.154216 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4f5\" (UniqueName: \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\") pod \"maas-api-revoke-sub-f45da628-lnpt6\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.164204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.164170 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:44.475594 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:44.475549 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8310579d_fbe3_4e6d_a050_5893d470d72f.slice/crio-13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6 WatchSource:0}: Error finding container 13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6: Status 404 returned error can't find the container with id 13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6 Oct 30 20:54:44.476270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.476246 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6"] Oct 30 20:54:44.518646 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.518612 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" event={"ID":"8310579d-fbe3-4e6d-a050-5893d470d72f","Type":"ContainerStarted","Data":"13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6"} Oct 30 20:54:44.519956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.519926 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" event={"ID":"fa927e25-3d99-40a7-a06c-b04601af6192","Type":"ContainerStarted","Data":"37bf48e6fca824d00e9cadc7eb7c540b82bddc9870f93eb5429214b6dc77c243"} Oct 30 20:54:44.521583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.521557 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" event={"ID":"412de240-9d58-4923-8926-6001b1d33f7d","Type":"ContainerStarted","Data":"78b68fe916f6e63c34b2c3b4bb4210e293fd65ed69fc56011558ec6c084aa8d2"} Oct 30 20:54:44.544344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:44.544296 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" podStartSLOduration=3.544269605 podStartE2EDuration="3.544269605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:44.540470054 +0000 UTC m=+897.899182800" watchObservedRunningTime="2026-10-30 20:54:44.544269605 +0000 UTC m=+897.902982353" Oct 30 20:54:45.612681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:45.612458 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" event={"ID":"8310579d-fbe3-4e6d-a050-5893d470d72f","Type":"ContainerStarted","Data":"9c65c2d2d076d9e8fe716cd676607d36b14b893e8c926b72589a602d43a2fb2d"} Oct 30 20:54:45.631686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:45.629826 5855 generic.go:356] "Generic (PLEG): container finished" podID="fa927e25-3d99-40a7-a06c-b04601af6192" containerID="1337268f2bb71982d97571b4fdd5127f13c983cf7d4a99421166e61bf46760b0" exitCode=0 Oct 30 20:54:45.631686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:45.630024 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" event={"ID":"fa927e25-3d99-40a7-a06c-b04601af6192","Type":"ContainerDied","Data":"1337268f2bb71982d97571b4fdd5127f13c983cf7d4a99421166e61bf46760b0"} Oct 30 20:54:45.644310 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:45.642119 5855 generic.go:356] "Generic (PLEG): container finished" podID="412de240-9d58-4923-8926-6001b1d33f7d" containerID="78b68fe916f6e63c34b2c3b4bb4210e293fd65ed69fc56011558ec6c084aa8d2" exitCode=0 Oct 30 20:54:45.644310 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:45.643238 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" event={"ID":"412de240-9d58-4923-8926-6001b1d33f7d","Type":"ContainerDied","Data":"78b68fe916f6e63c34b2c3b4bb4210e293fd65ed69fc56011558ec6c084aa8d2"} Oct 30 20:54:46.621352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.621312 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 20:54:46.627564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.627509 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.646633 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.646604 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 20:54:46.656278 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.656249 5855 generic.go:356] "Generic (PLEG): container finished" podID="8310579d-fbe3-4e6d-a050-5893d470d72f" containerID="9c65c2d2d076d9e8fe716cd676607d36b14b893e8c926b72589a602d43a2fb2d" exitCode=0 Oct 30 20:54:46.656433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.656313 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" event={"ID":"8310579d-fbe3-4e6d-a050-5893d470d72f","Type":"ContainerDied","Data":"9c65c2d2d076d9e8fe716cd676607d36b14b893e8c926b72589a602d43a2fb2d"} Oct 30 20:54:46.714701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.714642 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.714892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.714771 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.714892 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.714855 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715008 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.714932 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715008 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.714988 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84zz\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.715023 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.715112 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.715143 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.715271 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.715253 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.866280 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.866242 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.866602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.866558 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84zz\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.866803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.866786 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.866989 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.866975 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.867093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.867078 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.867301 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.867284 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.867519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.867487 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.867689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.867648 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.867883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.867864 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.871512 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.870153 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.871512 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.870744 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.871512 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.871426 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.872017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.871729 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:46.994419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:46.973173 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.013681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.007289 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:54:47.013681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.011718 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.013681 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:54:47.012491 5855 info.go:109] Failed to get network devices: open /sys/class/net/37bf48e6fca824d/address: no such file or directory Oct 30 20:54:47.027985 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.027955 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.056729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.050909 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:54:47.060880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.058380 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.060880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.058470 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:54:47.075321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.075291 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84zz\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.075542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.075523 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:54:47.122992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.122916 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.124126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.123431 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.129421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.128746 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 20:54:47.143085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.142750 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 20:54:47.143085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.142800 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.175369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.175325 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.175692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.175641 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.175949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.175929 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176319 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176404 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntdd\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176450 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176514 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176533 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.176738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176569 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176813 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176850 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176897 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176919 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.176986 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.177126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.177097 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.178137 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.178109 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.178228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.178168 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwvp\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.178228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.178213 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.196489 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.196464 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:47.199185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.199155 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:47.208868 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.208839 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:47.250192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.250164 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.279815 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\") pod \"fa927e25-3d99-40a7-a06c-b04601af6192\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.279875 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\" (UniqueName: \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\") pod \"fa927e25-3d99-40a7-a06c-b04601af6192\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.279898 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\") pod \"8310579d-fbe3-4e6d-a050-5893d470d72f\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280058 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\") pod \"412de240-9d58-4923-8926-6001b1d33f7d\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280106 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\" (UniqueName: \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\") pod \"412de240-9d58-4923-8926-6001b1d33f7d\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280175 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\") pod \"fa927e25-3d99-40a7-a06c-b04601af6192\" (UID: \"fa927e25-3d99-40a7-a06c-b04601af6192\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280210 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\") pod \"412de240-9d58-4923-8926-6001b1d33f7d\" (UID: \"412de240-9d58-4923-8926-6001b1d33f7d\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280317 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\" (UniqueName: \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\") pod \"8310579d-fbe3-4e6d-a050-5893d470d72f\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280362 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\") pod \"8310579d-fbe3-4e6d-a050-5893d470d72f\" (UID: \"8310579d-fbe3-4e6d-a050-5893d470d72f\") " Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280617 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280681 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280714 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwvp\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280739 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280838 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.280951 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281014 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281096 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281139 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281185 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntdd\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281217 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281290 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281315 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281340 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281385 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281448 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281492 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281513 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281550 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca" pod "fa927e25-3d99-40a7-a06c-b04601af6192" (UID: "fa927e25-3d99-40a7-a06c-b04601af6192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281558 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.281627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281614 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281695 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281739 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281769 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281792 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281831 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.281849 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.282199 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.282490 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.282779 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca" pod "8310579d-fbe3-4e6d-a050-5893d470d72f" (UID: "8310579d-fbe3-4e6d-a050-5893d470d72f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283094 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca" pod "412de240-9d58-4923-8926-6001b1d33f7d" (UID: "412de240-9d58-4923-8926-6001b1d33f7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283196 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283249 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhlq\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.283377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283322 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.284027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283530 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.284027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.283813 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.284027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.284017 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.284247 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.284223 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.285318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.284920 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.285318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.284963 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.285318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.285143 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.285318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.285168 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.285600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.285534 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.289219 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.289193 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.290674 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.290617 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5" pod "8310579d-fbe3-4e6d-a050-5893d470d72f" (UID: "8310579d-fbe3-4e6d-a050-5893d470d72f"). InnerVolumeSpecName "kube-api-access-9w4f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:47.290674 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.290631 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.292046 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.291934 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca" pod "8310579d-fbe3-4e6d-a050-5893d470d72f" (UID: "8310579d-fbe3-4e6d-a050-5893d470d72f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:47.292046 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.292010 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75" pod "412de240-9d58-4923-8926-6001b1d33f7d" (UID: "412de240-9d58-4923-8926-6001b1d33f7d"). InnerVolumeSpecName "kube-api-access-sft75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:47.292328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.292287 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca" pod "fa927e25-3d99-40a7-a06c-b04601af6192" (UID: "fa927e25-3d99-40a7-a06c-b04601af6192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:47.292413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.292375 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f" pod "fa927e25-3d99-40a7-a06c-b04601af6192" (UID: "fa927e25-3d99-40a7-a06c-b04601af6192"). InnerVolumeSpecName "kube-api-access-65w4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:47.293036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.292986 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.293626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.293594 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.294417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.294391 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca" pod "412de240-9d58-4923-8926-6001b1d33f7d" (UID: "412de240-9d58-4923-8926-6001b1d33f7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:47.297487 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.297467 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwvp\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.302455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.302421 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.303223 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.303200 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntdd\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.303413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.303397 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.387739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387607 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.387922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387772 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.387922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387878 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.387922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387912 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387944 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.387996 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhlq\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388081 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388116 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388333 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388172 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388333 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388271 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388333 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388281 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.388645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388616 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.389006 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.388982 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.389118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389048 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w4f5\" (UniqueName: \"kubernetes.io/projected/8310579d-fbe3-4e6d-a050-5893d470d72f-kube-api-access-9w4f5\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389068 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa927e25-3d99-40a7-a06c-b04601af6192-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389082 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65w4f\" (UniqueName: \"kubernetes.io/projected/fa927e25-3d99-40a7-a06c-b04601af6192-kube-api-access-65w4f\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389098 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8310579d-fbe3-4e6d-a050-5893d470d72f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389113 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412de240-9d58-4923-8926-6001b1d33f7d-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389379 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389128 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sft75\" (UniqueName: \"kubernetes.io/projected/412de240-9d58-4923-8926-6001b1d33f7d-kube-api-access-sft75\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.389794 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.389645 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.391488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.391457 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.391589 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.391463 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.400375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.400344 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.401959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.401929 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhlq\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.448158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.448117 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:47.470029 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.469988 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:47.509536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.508759 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:47.562563 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:47.562501 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5f684_595f_40e1_8549_c8a3b48e2866.slice/crio-d2b7389e9a49d5a8a43c3ac4c277e70b7bd9bb8bd08da0819979546d09804eb2 WatchSource:0}: Error finding container d2b7389e9a49d5a8a43c3ac4c277e70b7bd9bb8bd08da0819979546d09804eb2: Status 404 returned error can't find the container with id d2b7389e9a49d5a8a43c3ac4c277e70b7bd9bb8bd08da0819979546d09804eb2 Oct 30 20:54:47.571101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.565915 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.571101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.565986 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.571101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.566025 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.571101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.568079 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 20:54:47.799974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.795477 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" event={"ID":"412de240-9d58-4923-8926-6001b1d33f7d","Type":"ContainerDied","Data":"fd0bc796d57040d04d4cf5b6bf6c511862fe1282b4db77f5dbe3bc2dd1f71fed"} Oct 30 20:54:47.799974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.795523 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0bc796d57040d04d4cf5b6bf6c511862fe1282b4db77f5dbe3bc2dd1f71fed" Oct 30 20:54:47.799974 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.795581 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7" Oct 30 20:54:47.816552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.814107 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" event={"ID":"8310579d-fbe3-4e6d-a050-5893d470d72f","Type":"ContainerDied","Data":"13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6"} Oct 30 20:54:47.816552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.814166 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a4b6427c62aeba24ed87f84426c8973fd8cc9ab0f69e2c93452c0938c40ee6" Oct 30 20:54:47.816552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.814284 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6" Oct 30 20:54:47.828520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.820152 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.828520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.820245 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.828520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.820288 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.828520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.827191 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:54:47.830473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.830372 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" event={"ID":"fa927e25-3d99-40a7-a06c-b04601af6192","Type":"ContainerDied","Data":"37bf48e6fca824d00e9cadc7eb7c540b82bddc9870f93eb5429214b6dc77c243"} Oct 30 20:54:47.830473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.830429 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bf48e6fca824d00e9cadc7eb7c540b82bddc9870f93eb5429214b6dc77c243" Oct 30 20:54:47.833866 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.830930 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt" Oct 30 20:54:47.839564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.839534 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" event={"ID":"99c5f684-595f-40e1-8549-c8a3b48e2866","Type":"ContainerStarted","Data":"d2b7389e9a49d5a8a43c3ac4c277e70b7bd9bb8bd08da0819979546d09804eb2"} Oct 30 20:54:47.899249 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:47.899208 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f057ef4_4650_4566_bf29_0df595a4c5f2.slice/crio-ab438e713866252413f81703a6a905ec900889d9534316c61bcbf6602380e34c WatchSource:0}: Error finding container ab438e713866252413f81703a6a905ec900889d9534316c61bcbf6602380e34c: Status 404 returned error can't find the container with id ab438e713866252413f81703a6a905ec900889d9534316c61bcbf6602380e34c Oct 30 20:54:47.900979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.900952 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 20:54:47.903307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.903267 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.903427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.903364 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:47.903427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:47.903408 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:48.032145 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:48.032029 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:54:48.036118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:48.035694 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:48.036118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:48.035788 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:48.036118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:48.035831 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:54:49.093431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.093391 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" event={"ID":"99c5f684-595f-40e1-8549-c8a3b48e2866","Type":"ContainerStarted","Data":"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1"} Oct 30 20:54:49.249690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.240072 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" event={"ID":"3f057ef4-4650-4566-bf29-0df595a4c5f2","Type":"ContainerStarted","Data":"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79"} Oct 30 20:54:49.250539 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.250498 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" event={"ID":"3f057ef4-4650-4566-bf29-0df595a4c5f2","Type":"ContainerStarted","Data":"ab438e713866252413f81703a6a905ec900889d9534316c61bcbf6602380e34c"} Oct 30 20:54:49.252453 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.252429 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:54:49.257687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.256792 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:49.257687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.256850 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:49.387775 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.387610 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" event={"ID":"72a54c98-8ed7-4691-b3f2-c6000eab4f87","Type":"ContainerStarted","Data":"e1be3b466fa587b1d64c76ab7bb201f68fe980f3b74ca3887663e93b9be73c00"} Oct 30 20:54:49.388086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.388063 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" event={"ID":"72a54c98-8ed7-4691-b3f2-c6000eab4f87","Type":"ContainerStarted","Data":"b243369bcb84af1b9360dbae2bf7ba0c5cfc40d28eb095d65365d2f0b9ce6a97"} Oct 30 20:54:49.488808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.488772 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:49.489136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.489108 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:49.489286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.488815 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:54:49.491644 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.491612 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" event={"ID":"55fb2495-6764-4bea-8c5e-905417748912","Type":"ContainerStarted","Data":"2a6248e0f32b171133e72e80f259cc26cc3427bc760cffbd7152a69aeda489b9"} Oct 30 20:54:49.491872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.491853 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" event={"ID":"55fb2495-6764-4bea-8c5e-905417748912","Type":"ContainerStarted","Data":"16a2844141cf67e7f9f4df553061ba63c3a6310f686e6160ff3170934ab4d4bf"} Oct 30 20:54:49.514508 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.514474 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:54:49.517128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.516645 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:49.517377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.517353 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:49.522990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.522912 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podStartSLOduration=3.522886255 podStartE2EDuration="3.522886255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:49.308462686 +0000 UTC m=+902.667175432" watchObservedRunningTime="2026-10-30 20:54:49.522886255 +0000 UTC m=+902.881599001" Oct 30 20:54:49.551334 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.551268 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podStartSLOduration=3.551246248 podStartE2EDuration="3.551246248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:49.484576987 +0000 UTC m=+902.843289741" watchObservedRunningTime="2026-10-30 20:54:49.551246248 +0000 UTC m=+902.909958996" Oct 30 20:54:49.553251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.553194 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podStartSLOduration=3.553177597 podStartE2EDuration="3.553177597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:49.431236418 +0000 UTC m=+902.789949171" watchObservedRunningTime="2026-10-30 20:54:49.553177597 +0000 UTC m=+902.911890340" Oct 30 20:54:49.622958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.622889 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podStartSLOduration=3.622863042 podStartE2EDuration="3.622863042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:49.532253603 +0000 UTC m=+902.890966349" watchObservedRunningTime="2026-10-30 20:54:49.622863042 +0000 UTC m=+902.981575789" Oct 30 20:54:49.887678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.878752 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:49.975874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:49.975822 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.033841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.033782 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:54:50.136890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.136826 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.254227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.254140 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:50.254596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.254550 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:50.452216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.452180 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:54:50.452596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.452575 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:50.463758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.456745 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:50.471115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.471079 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:50.471508 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.471470 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:50.496682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.494891 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.523842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.523611 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:50.523842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.523689 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:50.559809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.559761 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.568735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.568694 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.585858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.585817 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:50.636808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.636754 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.705970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.705919 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.706634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.706604 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.976806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.976757 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.981410 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.981362 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:50.983234 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:50.981975 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:51.053681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.034895 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.163837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.163803 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:51.299748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.292642 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:51.299748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.292725 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:51.299748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.292799 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:54:51.322238 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.320413 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:51.369684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.366741 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:51.480695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.480282 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.498855 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.498909 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.500077 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.507484 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.507773 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.512299 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:51.581747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.512338 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.582918 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.583016 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.583594 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.583647 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.592000 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.592055 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.593146 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:51.596964 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.593408 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:51.629770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.625226 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.720557 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.720519 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:51.722188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.722164 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:54:51.798681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.773796 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:51.889264 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:51.889155 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:52.024679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.006373 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:52.026871 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.026842 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:52.146125 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.145342 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:52.162717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.155810 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:54:52.231884 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.231837 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:52.233277 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.233252 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:52.355681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.354924 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:52.355681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.355019 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:52.355681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.355273 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.495525 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.495607 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.512563 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.513641 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.536943 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.513270 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.578964 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.579034 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.588961 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:52.620027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.589630 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:52.650677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.636833 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:52.650677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.636928 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:52.844241 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.764790 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 30 20:54:52.844241 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:52.798879 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.121480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.121326 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.133837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.121758 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded" Oct 30 20:54:53.133837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.123198 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.147606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.147560 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj"] Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148533 5855 cpu_manager.go:414] "containerMap: removing container" podUID="412de240-9d58-4923-8926-6001b1d33f7d" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148569 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412de240-9d58-4923-8926-6001b1d33f7d" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148583 5855 cpu_manager.go:414] "containerMap: removing container" podUID="fa927e25-3d99-40a7-a06c-b04601af6192" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148590 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa927e25-3d99-40a7-a06c-b04601af6192" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148626 5855 cpu_manager.go:414] "containerMap: removing container" podUID="8310579d-fbe3-4e6d-a050-5893d470d72f" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148634 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8310579d-fbe3-4e6d-a050-5893d470d72f" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148822 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="412de240-9d58-4923-8926-6001b1d33f7d" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148834 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa927e25-3d99-40a7-a06c-b04601af6192" containerName="revoke-keys" Oct 30 20:54:53.155682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.148848 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8310579d-fbe3-4e6d-a050-5893d470d72f" containerName="revoke-keys" Oct 30 20:54:53.164688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.161769 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.201682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.180356 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.211680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.209403 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:54:53.211680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.210238 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:54:53.211680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.210308 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:53.211680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.210386 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:54:53.245172 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.235783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.254187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.254149 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj"] Oct 30 20:54:53.254865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.254836 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:53.264152 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.264110 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:53.284712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.282135 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.284712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.282475 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlz6\" (UniqueName: \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.284712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.282626 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.367847 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.367770 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.400791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.384285 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlz6\" (UniqueName: \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.400791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.384421 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.400791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.384567 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.402640 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.402534 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.417220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.417150 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.429576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.429492 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlz6\" (UniqueName: \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\") pod \"maas-api-revoke-sub-971d237a-xszmj\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.449543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.449091 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:53.449543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.449156 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:53.474699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.474619 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:53.474882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.474706 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:53.506683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.497108 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.538717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.520120 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:53.538717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.520184 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:53.538717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.521749 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:53.538717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.522118 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:53.581752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.575473 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.581752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.575969 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:53.656300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.593808 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.656300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.594180 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:53.656300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.597514 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:54:53.852680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.797041 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.852680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.809052 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:53.982739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:53.915103 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:54.133681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.128814 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:54.133681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.129248 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:54.133681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.129935 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.187244 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.241845 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.254600 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.254672 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.376699 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.448536 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:54.451948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.448593 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:54.499445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.474050 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:54.499445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.474108 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:54.528573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.520414 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:54.528573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.520474 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:54.536132 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.530984 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn"] Oct 30 20:54:54.557867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.555925 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:54.557867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.556270 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.583496 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.593988 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.594763 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.818773 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.820229 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.024895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:54.917521 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.216681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.184866 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:54:55.216681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.188328 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.221827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.220614 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:55.247921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.246737 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:55.387150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.279827 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:55.387150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.279891 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:55.388815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.387873 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:55.538784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.537968 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:55.538784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.538062 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:55.538784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.538149 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:55.723686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.722848 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:54:55.723686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.723315 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:55.790199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.767984 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn"] Oct 30 20:54:55.790199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.768192 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:54:55.790199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.768259 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:54:55.835689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.833342 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:55.835689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.833433 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:55.865752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.859459 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:56.006840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.006721 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:56.008501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.983506 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.008726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.990221 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:54:56.009046 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.982330 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.016900 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:55.990328 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:56.017171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.017142 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:56.038535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.029139 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:56.038535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.029208 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:56.038535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.035307 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:54:56.048728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.046512 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.048728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.046631 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.067945 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.066392 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:56.174993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.174951 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.205127 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.205094 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:56.567874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.551318 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:56.912838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.537333 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:54:56.913204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.537396 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:56.913395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.537440 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:56.913555 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.429882 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:56.913792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.913747 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:56.913955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.767017 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:56.946134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.813315 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:56.965724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.484407 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:56.965724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.949374 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:56.965724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.609516 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\") pod \"maas-api-revoke-sub-36101384-mggkn\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:56.997248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:56.997201 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.004473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.003238 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.026439 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.025050 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:54:57.054822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.054779 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.149182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.149138 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.161713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.161055 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.575690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.571906 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.632477 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.613123 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.633172 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.633130 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:57.644792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.628185 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:54:57.644792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.641967 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj"] Oct 30 20:54:57.666399 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.666300 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:54:57.667467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.667423 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.686829 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.668732 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.686948 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.669758 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.687009 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:57.689686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.576292 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:57.744174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.744124 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:57.839171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.839136 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:54:57.839473 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.839449 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" event={"ID":"bea05bb9-9243-408c-8a9a-fa3cd1600892","Type":"ContainerStarted","Data":"f5111952b2d2e0eb8d6db8e3c95284962f88f227a539db746463f121b0462bee"} Oct 30 20:54:57.922906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.921906 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:54:57.922906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.922024 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:54:57.923205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.879638 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:57.923205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:57.923143 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:58.092771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.092726 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:58.107129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.106971 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:54:58.114343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.114294 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:54:58.117685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.114893 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 30 20:54:58.161370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.161308 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:58.173880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.173769 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:58.174557 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.174534 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:58.181599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.181542 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:58.264684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.263152 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:58.264684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.263208 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:58.484772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.478735 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:58.484772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.478801 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:58.496697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.486386 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:58.496697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.486441 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:58.496697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.486963 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:54:58.516070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.511542 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:58.536582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.516580 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:58.573835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.573799 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:54:58.656712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.656641 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:58.863815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.862930 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:54:58.867690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.864355 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:58.973681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:58.971235 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.039840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.039785 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.114828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.113683 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.126846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.115086 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.186867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.186814 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.242833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.202803 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:54:59.242833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.208997 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:54:59.536740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.536143 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5"] Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.617393 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.617445 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.619256 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.619308 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.619379 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.637591 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.637646 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.642042 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.642140 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:54:59.682456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.661070 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:59.684153 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.187382 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:54:59.684972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.684943 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:54:59.685142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.685116 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:54:59.721865 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:54:59.705326 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cca3ea5_6879_4af0_a616_9f678f01e951.slice/crio-34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9 WatchSource:0}: Error finding container 34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9: Status 404 returned error can't find the container with id 34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9 Oct 30 20:54:59.738608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.736054 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5"] Oct 30 20:54:59.759696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.756569 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:54:59.789242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.753782 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:54:59.852707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.847204 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:54:59.853165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.853136 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:54:59.853400 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.853384 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll57z\" (UniqueName: \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:54:59.853928 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.848345 5855 generic.go:356] "Generic (PLEG): container finished" podID="bea05bb9-9243-408c-8a9a-fa3cd1600892" containerID="edd0a0337dbc741ce7033964782d03709b951e2a80a7e0d3a20ef540252ced65" exitCode=0 Oct 30 20:54:59.865681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.848382 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" event={"ID":"bea05bb9-9243-408c-8a9a-fa3cd1600892","Type":"ContainerDied","Data":"edd0a0337dbc741ce7033964782d03709b951e2a80a7e0d3a20ef540252ced65"} Oct 30 20:54:59.866425 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:54:59.866350 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.001268 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.001216 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:00.025680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.007447 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:00.025680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.007496 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll57z\" (UniqueName: \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:00.025680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.002645 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:00.025680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.002726 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:00.025680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.012042 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:55:00.362680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.362108 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:00.362970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.362931 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.407682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.372871 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.407682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.402290 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 20:55:00.407682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.402570 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:55:00.408105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.408068 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:00.462431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.462392 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:00.464950 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.463421 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll57z\" (UniqueName: \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:00.502228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.494258 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:55:00.502228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.494340 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:55:00.538685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.517304 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:55:00.538685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.517396 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:55:00.538685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.520064 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:00.538685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.520120 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:00.545737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.543088 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:00.633149 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.633102 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.633918 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.633895 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:55:00.671942 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.671889 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:00.697689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.683838 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.708834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.707847 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.716020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.709808 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:00.716020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.709911 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:00.716020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.710748 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:55:00.882742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.874176 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.950151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:00.950117 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c399b28-v8bj5\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:01.071815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.071760 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.210278 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.210248 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:01.445855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.437866 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:01.507769 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.474870 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.516687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.503671 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:01.529688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.521554 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.643867 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.595630 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.626887 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.595233 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.595572 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.647064 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.595987 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:55:01.665756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.643835 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:01.730602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.655257 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:55:01.730602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.680591 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.730602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.693028 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.843354 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:55:01.768787 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:55:01.843354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.663101 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:01.843354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.839922 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:01.843354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.814825 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:01.843354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.839827 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:01.953698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.925058 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.953698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.925412 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:01.953698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:01.925681 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 20:55:02.240682 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:55:02.225739 5855 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.405s" Oct 30 20:55:02.240682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.225839 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:55:02.240682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.225896 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:55:02.240682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.225914 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" event={"ID":"0cca3ea5-6879-4af0-a616-9f678f01e951","Type":"ContainerStarted","Data":"34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9"} Oct 30 20:55:02.240682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.226039 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:55:02.250941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.250897 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.256854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.253780 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:55:02.397850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.387990 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:55:02.564449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.564417 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:55:02.564820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.564799 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:55:02.564971 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.564959 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:55:02.584008 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.467984 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:55:02.613851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.613812 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:55:02.614101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.515752 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:02.635684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.515693 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.637184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.521786 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.637476 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.482731 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:02.637671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.637619 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:02.637823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.515609 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:55:02.637943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.637921 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:55:02.643103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.643066 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" event={"ID":"0cca3ea5-6879-4af0-a616-9f678f01e951","Type":"ContainerStarted","Data":"5d95b19a84065f5a99a96ae15e86ab39cdb3b65659b84772e4d3e3f2372d2096"} Oct 30 20:55:02.691546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.644103 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.702292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.648825 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.803187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.761347 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:02.845959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.837804 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:02.845959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.837868 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.847443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.846371 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.847443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.846819 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:02.847443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.847120 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 30 20:55:02.847443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.847173 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:02.878755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.869624 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.942565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.942429 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.017708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:02.954735 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:03.137802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.127796 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.282680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.252027 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.517457 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.517341 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.525797 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.525738 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:03.550772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.548348 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 20:55:03.550772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.548419 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 20:55:03.551183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.526091 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:03.552065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.552036 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:55:03.552356 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.552333 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:55:03.621988 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.621950 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:03.700354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.700305 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.702010 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.701965 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:03.728867 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.728810 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.763726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.739444 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn"] Oct 30 20:55:03.783760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.770064 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:55:03.830202 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.830164 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:03.873161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.873119 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.873619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.873592 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:03.873848 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.873816 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:03.877225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.874020 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.877225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.874065 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:03.919975 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.919935 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:03.920231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:03.920207 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.045745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.020627 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.246083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.246046 5855 generic.go:356] "Generic (PLEG): container finished" podID="0cca3ea5-6879-4af0-a616-9f678f01e951" containerID="5d95b19a84065f5a99a96ae15e86ab39cdb3b65659b84772e4d3e3f2372d2096" exitCode=0 Oct 30 20:55:04.255079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.255041 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" event={"ID":"0cca3ea5-6879-4af0-a616-9f678f01e951","Type":"ContainerDied","Data":"5d95b19a84065f5a99a96ae15e86ab39cdb3b65659b84772e4d3e3f2372d2096"} Oct 30 20:55:04.313752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.289793 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.578850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.578747 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:04.579143 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.579091 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:04.626351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.626316 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:55:04.667726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.653079 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.704208 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.704166 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:04.714680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.704941 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:04.886874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.886757 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.943990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.943944 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:04.960068 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.909482 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:04.960569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.909501 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:04.989040 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.909440 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Oct 30 20:55:05.052424 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.962065 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:05.053079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:04.984862 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:05.080727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.015850 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.080727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.061807 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:05.080727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.079980 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.106111 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.106054 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.418110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.418068 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:05.495645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.471743 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:05.495645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.471805 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:05.518479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.517366 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:05.518479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.517430 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.562692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.562573 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:05.562692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.562636 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.628295 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.628255 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.715757 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.698425 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.717387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.717330 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.750500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.718542 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.986751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.958891 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.986751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.960342 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.986751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:05.963980 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.028925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.028741 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.158680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.142184 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.159874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.159801 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:06.173981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.173924 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.213981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.207218 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.213981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.207329 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:06.213981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.207371 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.213981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.207439 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.286733 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.222830 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:55:06.507880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.503538 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.619032 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.618511 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 30 20:55:06.660909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.659916 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:06.660909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.659995 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.670678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.670613 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:06.671685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.671642 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:55:06.696740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.674860 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:06.696740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.694386 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:06.696740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.694091 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:06.696740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.694763 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.730721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.720272 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:06.730721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.720707 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:06.730721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.721016 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:07.002685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:06.999048 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5"] Oct 30 20:55:07.243834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.061825 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:07.353251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.061894 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.108043 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.120803 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.221949 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.223209 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.247999 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.248074 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.265018 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:07.436898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.423683 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.461101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.457195 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:55:07.493684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.471013 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 20:55:07.493684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.471130 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 20:55:07.517751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.511799 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.638711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.637349 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" event={"ID":"534a4532-6a64-4875-bfa9-58a7ed15edc9","Type":"ContainerStarted","Data":"c5d10c783448b188bf46b1cec37fbdc675d48699365b7e45060b7c6120f5869e"} Oct 30 20:55:07.730689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.729764 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.734771 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.735312 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.735890 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.737444 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\" (UniqueName: \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\") pod \"bea05bb9-9243-408c-8a9a-fa3cd1600892\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.737540 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\") pod \"bea05bb9-9243-408c-8a9a-fa3cd1600892\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " Oct 30 20:55:07.738690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.737695 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\") pod \"bea05bb9-9243-408c-8a9a-fa3cd1600892\" (UID: \"bea05bb9-9243-408c-8a9a-fa3cd1600892\") " Oct 30 20:55:07.811682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.811022 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:55:07.811682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.811178 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:07.811682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.811233 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.836021 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.814331 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:07.836021 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.814407 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.836021 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.833887 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca" pod "bea05bb9-9243-408c-8a9a-fa3cd1600892" (UID: "bea05bb9-9243-408c-8a9a-fa3cd1600892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:07.871900 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.855337 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\") pod \"0cca3ea5-6879-4af0-a616-9f678f01e951\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " Oct 30 20:55:07.911922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.885219 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\") pod \"0cca3ea5-6879-4af0-a616-9f678f01e951\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " Oct 30 20:55:07.911922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.910817 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\") pod \"0cca3ea5-6879-4af0-a616-9f678f01e951\" (UID: \"0cca3ea5-6879-4af0-a616-9f678f01e951\") " Oct 30 20:55:07.935353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:07.935289 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca" pod "0cca3ea5-6879-4af0-a616-9f678f01e951" (UID: "0cca3ea5-6879-4af0-a616-9f678f01e951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:08.039688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.008838 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6" pod "bea05bb9-9243-408c-8a9a-fa3cd1600892" (UID: "bea05bb9-9243-408c-8a9a-fa3cd1600892"). InnerVolumeSpecName "kube-api-access-2vlz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:08.039688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.008946 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca" pod "bea05bb9-9243-408c-8a9a-fa3cd1600892" (UID: "bea05bb9-9243-408c-8a9a-fa3cd1600892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:08.039688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.009016 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx" pod "0cca3ea5-6879-4af0-a616-9f678f01e951" (UID: "0cca3ea5-6879-4af0-a616-9f678f01e951"). InnerVolumeSpecName "kube-api-access-fkwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:08.079586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.079550 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vlz6\" (UniqueName: \"kubernetes.io/projected/bea05bb9-9243-408c-8a9a-fa3cd1600892-kube-api-access-2vlz6\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.506796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.506758 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/0cca3ea5-6879-4af0-a616-9f678f01e951-kube-api-access-fkwxx\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.532112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.532082 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.532405 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.532385 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea05bb9-9243-408c-8a9a-fa3cd1600892-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.532541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.532528 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.532642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.377187 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" event={"ID":"bea05bb9-9243-408c-8a9a-fa3cd1600892","Type":"ContainerDied","Data":"f5111952b2d2e0eb8d6db8e3c95284962f88f227a539db746463f121b0462bee"} Oct 30 20:55:08.532805 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.532786 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5111952b2d2e0eb8d6db8e3c95284962f88f227a539db746463f121b0462bee" Oct 30 20:55:08.532919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.446985 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:08.543297 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.447033 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:08.551686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.447054 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:08.555687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.447073 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:08.572740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.447245 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:08.609232 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.447271 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.516021 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.146768 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.214906 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca" pod "0cca3ea5-6879-4af0-a616-9f678f01e951" (UID: "0cca3ea5-6879-4af0-a616-9f678f01e951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.490180 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.613221 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.359151 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.359190 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.636684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.395814 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj" Oct 30 20:55:08.693587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.693490 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cca3ea5-6879-4af0-a616-9f678f01e951-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:08.744591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.744551 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.744915 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.744881 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:08.759719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.755802 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.765114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.764021 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:08.865724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.858781 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:08.865724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.858847 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.923873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.923841 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:08.956994 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:08.956897 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.087683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.079234 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.533685 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:55:09.515357 5855 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea05bb9_9243_408c_8a9a_fa3cd1600892.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea05bb9_9243_408c_8a9a_fa3cd1600892.slice/crio-f5111952b2d2e0eb8d6db8e3c95284962f88f227a539db746463f121b0462bee\": RecentStats: unable to find data in memory cache]" Oct 30 20:55:09.589061 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.561071 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:09.589419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.589376 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.589945 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.567366 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:09.590557 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.567406 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:09.616868 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.582934 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.617406 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.533760 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.617742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.567427 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:09.620829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.619780 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.620829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.619897 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.620829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.620200 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:09.620829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.620479 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.697889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.645373 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:09.697889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.645496 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.697889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.647100 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.697889 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:55:09.674458 5855 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea05bb9_9243_408c_8a9a_fa3cd1600892.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cca3ea5_6879_4af0_a616_9f678f01e951.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea05bb9_9243_408c_8a9a_fa3cd1600892.slice/crio-f5111952b2d2e0eb8d6db8e3c95284962f88f227a539db746463f121b0462bee\": RecentStats: unable to find data in memory cache]" Oct 30 20:55:09.724697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.708783 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" event={"ID":"534a4532-6a64-4875-bfa9-58a7ed15edc9","Type":"ContainerStarted","Data":"24e0a733a6c43eb9b81650cf679452e34f113656a472eb1bdc6b01c64d46001f"} Oct 30 20:55:09.772685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.752779 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:09.772685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.755057 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:09.773208 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.773174 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:09.806684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.804141 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" event={"ID":"0cca3ea5-6879-4af0-a616-9f678f01e951","Type":"ContainerDied","Data":"34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9"} Oct 30 20:55:09.806684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.804206 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34fe190a022e7c906c8d5daa8efc06d64fe640661c1823d1a4bd78513d26cec9" Oct 30 20:55:09.806684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.804279 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn" Oct 30 20:55:09.827689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.808758 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.881683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.878005 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:09.881991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.881953 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.984456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.984356 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:09.984745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:09.984711 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.072830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.072797 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.073081 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.073058 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.156685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.123485 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.344243 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.344161 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" podStartSLOduration=13.344138833 podStartE2EDuration="13.344138833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:09.759889705 +0000 UTC m=+923.118602487" watchObservedRunningTime="2026-10-30 20:55:10.344138833 +0000 UTC m=+923.702851582" Oct 30 20:55:10.601746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.597875 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:10.631813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.631756 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.631813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.631813 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.639694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.632389 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:10.639694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.632647 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:10.664726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.662776 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:10.664726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.663193 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:10.664726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.663478 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:10.681687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.681013 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:10.730763 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.730263 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:55:10.741063 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.741022 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.760973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.759592 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.761689 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.759627 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:10.762378 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.761346 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.773789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.773744 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.795752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.776811 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.796620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.796584 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.879925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.879805 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:10.890732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.889286 5855 generic.go:356] "Generic (PLEG): container finished" podID="534a4532-6a64-4875-bfa9-58a7ed15edc9" containerID="24e0a733a6c43eb9b81650cf679452e34f113656a472eb1bdc6b01c64d46001f" exitCode=0 Oct 30 20:55:10.890732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.889390 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" event={"ID":"534a4532-6a64-4875-bfa9-58a7ed15edc9","Type":"ContainerDied","Data":"24e0a733a6c43eb9b81650cf679452e34f113656a472eb1bdc6b01c64d46001f"} Oct 30 20:55:10.912459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.911773 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.912459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.911842 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.922846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.922804 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.923151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.923121 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:10.923526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.922819 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:10.923741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:10.923719 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.028852 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.028814 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.029174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.029140 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.100895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.100855 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.101206 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.101175 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.286890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.286787 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.287485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.287452 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.441833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.441779 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.619681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.618817 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.651092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.632606 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.651092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.633479 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.651092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.633783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:11.651092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.634019 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 20:55:11.651092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.634067 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:11.669564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.666939 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.669564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.667346 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.669564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.667636 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.684917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.684861 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:11.764803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.764749 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:11.768196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.766724 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.808321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.807784 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.809202 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.809171 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.809453 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.809427 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.852976 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.852938 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.853252 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.853228 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:11.885769 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.884751 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:11.920548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.919846 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:11.920548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:11.919920 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.148101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.148002 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.148380 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.148348 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.149065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.149041 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:55:12.149238 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.149219 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:55:12.591188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.590794 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.591188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.590870 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.621346 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.620769 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.636687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.635791 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.636687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.636219 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:12.636687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.636512 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:12.637098 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.636818 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:12.667788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.667743 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:12.668501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.668045 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.670052 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.668103 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.685924 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.685874 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.750984 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.750934 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.768834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.768779 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.769507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.769467 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.811836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.811797 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.859925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.859822 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.860271 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.860234 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.889375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.889327 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.901103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.897764 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.901103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.897829 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.942344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.941849 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.942344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.941955 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:12.972526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.972492 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:12.972875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:12.972829 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.182198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.182098 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:55:13.201788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.201738 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:55:13.202096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.199029 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.350859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.350802 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:13.351316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.351276 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.388976 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7"] Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.389950 5855 cpu_manager.go:414] "containerMap: removing container" podUID="bea05bb9-9243-408c-8a9a-fa3cd1600892" containerName="revoke-keys" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.389973 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea05bb9-9243-408c-8a9a-fa3cd1600892" containerName="revoke-keys" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.390061 5855 cpu_manager.go:414] "containerMap: removing container" podUID="0cca3ea5-6879-4af0-a616-9f678f01e951" containerName="revoke-keys" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.390071 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cca3ea5-6879-4af0-a616-9f678f01e951" containerName="revoke-keys" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.390261 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cca3ea5-6879-4af0-a616-9f678f01e951" containerName="revoke-keys" Oct 30 20:55:13.392531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.390275 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea05bb9-9243-408c-8a9a-fa3cd1600892" containerName="revoke-keys" Oct 30 20:55:13.447672 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.446493 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:13.448590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.448537 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.480682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.472501 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7"] Oct 30 20:55:13.538858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.538814 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:13.539175 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.539143 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.628630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.628580 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:13.649186 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.649139 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.660933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.649736 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:13.661540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.650847 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:13.673441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.650876 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:13.775796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.696804 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.775796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.740334 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:13.775796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.740714 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.790864 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.792030 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r226l\" (UniqueName: \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.792168 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.792377 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.826147 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.827058 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:13.829167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.827320 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:13.893680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.891886 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.912891 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.932220 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r226l\" (UniqueName: \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.931885 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.938974 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.943886 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.944037 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.979254 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.989886 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:13.989948 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.033828 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:14.034592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.033894 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.229076 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.225438 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r226l\" (UniqueName: \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\") pod \"maas-api-revoke-sub-3bf5d95c-ctdw7\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.233385 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.232987 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:14.274459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.273165 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:14.274459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.273225 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.553092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.553014 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:14.666989 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.665637 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.668982 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\") pod \"534a4532-6a64-4875-bfa9-58a7ed15edc9\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.669177 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\") pod \"534a4532-6a64-4875-bfa9-58a7ed15edc9\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.669309 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\" (UniqueName: \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\") pod \"534a4532-6a64-4875-bfa9-58a7ed15edc9\" (UID: \"534a4532-6a64-4875-bfa9-58a7ed15edc9\") " Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.695770 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.696116 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.697835 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.698055 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.698220 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:14.710820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.698951 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca" pod "534a4532-6a64-4875-bfa9-58a7ed15edc9" (UID: "534a4532-6a64-4875-bfa9-58a7ed15edc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:14.752724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.740748 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.752724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.741243 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:14.768889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.757396 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:14.817697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.807441 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.817697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.811935 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z" pod "534a4532-6a64-4875-bfa9-58a7ed15edc9" (UID: "534a4532-6a64-4875-bfa9-58a7ed15edc9"). InnerVolumeSpecName "kube-api-access-ll57z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:14.829026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.826821 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:14.829026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.827931 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:14.829026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.827975 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:14.829026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.827924 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca" pod "534a4532-6a64-4875-bfa9-58a7ed15edc9" (UID: "534a4532-6a64-4875-bfa9-58a7ed15edc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:14.829026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.828222 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Oct 30 20:55:15.130822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.885365 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534a4532-6a64-4875-bfa9-58a7ed15edc9-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:15.130822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.885403 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll57z\" (UniqueName: \"kubernetes.io/projected/534a4532-6a64-4875-bfa9-58a7ed15edc9-kube-api-access-ll57z\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:15.130822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:14.989677 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:15.163608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.150777 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:15.163608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.150850 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.166644 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.164561 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:15.166644 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.164704 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.212166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.211437 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:15.212166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.211498 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.330474 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.329785 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:15.330474 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.329849 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.482524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.469207 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" event={"ID":"534a4532-6a64-4875-bfa9-58a7ed15edc9","Type":"ContainerDied","Data":"c5d10c783448b188bf46b1cec37fbdc675d48699365b7e45060b7c6120f5869e"} Oct 30 20:55:15.482524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.469272 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5d10c783448b188bf46b1cec37fbdc675d48699365b7e45060b7c6120f5869e" Oct 30 20:55:15.482524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.469343 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5" Oct 30 20:55:15.521870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.521838 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm"] Oct 30 20:55:15.523242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.523215 5855 cpu_manager.go:414] "containerMap: removing container" podUID="534a4532-6a64-4875-bfa9-58a7ed15edc9" containerName="revoke-keys" Oct 30 20:55:15.540605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.540540 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a4532-6a64-4875-bfa9-58a7ed15edc9" containerName="revoke-keys" Oct 30 20:55:15.541159 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.541138 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a4532-6a64-4875-bfa9-58a7ed15edc9" containerName="revoke-keys" Oct 30 20:55:15.565955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.565924 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:15.646051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.601636 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm"] Oct 30 20:55:15.667077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.667033 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.703533 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.703489 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.704185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.704149 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:15.704643 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.704619 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:15.732766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.708696 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:15.732766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.709061 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:15.732766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.728771 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g58k\" (UniqueName: \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:15.732766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.728868 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:15.734364 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.734267 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.741047 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.765926 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.837775 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.837818 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.837845 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:15.923881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.882808 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:15.995968 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:15.994812 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.118715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.022739 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g58k\" (UniqueName: \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.118715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.022819 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.118715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.022937 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.118715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.024147 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.169684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.166636 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:16.169684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.166747 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.169684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.167288 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.432266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.178182 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.432266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.187830 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:16.432266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.187897 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.432266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.193473 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g58k\" (UniqueName: \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\") pod \"maas-api-revoke-sub-9e18f318-mggdm\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.665182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.665152 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:55:16.668377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.666726 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:55:16.676235 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.675485 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.709514 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.709430 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 20:55:16.715133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.715089 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:16.716973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.716939 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.718721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.715179 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:16.718956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.718914 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.721526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.721496 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:16.722754 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.722728 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:16.734883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.734074 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.737161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.735350 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:16.751099 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.741388 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 30 20:55:16.797262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.797207 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:16.829936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.823164 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.939802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.939092 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:16.971809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.949211 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:16.971809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.949960 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:16.971809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.950786 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:16.971809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:16.956363 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:17.086999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.086967 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:55:17.286880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.286798 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:17.286880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.286861 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.513963 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.513872 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:55:17.598829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.581308 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:55:17.598829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.581370 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7"] Oct 30 20:55:17.648959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.648925 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:55:17.685714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.659786 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:17.685714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.663000 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 20:55:17.689401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.686783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.732735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.722673 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.732735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.723031 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:17.745696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.735464 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.827682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.826216 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:55:17.827682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.827122 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.838615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.828799 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:55:17.838615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.836982 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:55:17.848401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.848373 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:55:17.868157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.850759 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:55:17.868157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.867013 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:17.993095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:17.993048 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:18.068680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.008482 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:55:18.068680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.034084 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:18.255554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.253798 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 30 20:55:18.275417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.257617 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:55:18.275417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.258532 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:55:18.275417 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.266246 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:18.301951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.300510 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr"] Oct 30 20:55:18.374813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.374783 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:55:18.375082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.375069 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:55:18.375237 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.375227 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:55:18.375922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.375907 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:55:18.376029 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.376018 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr"] Oct 30 20:55:18.376162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.376139 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" event={"ID":"006919cd-370a-4fef-9ffa-ca267d4c54c4","Type":"ContainerStarted","Data":"fe83f8039e56c33a394a3e1cd726b8a471e8a882aa69cc8c56ecb7f6c9d4f974"} Oct 30 20:55:18.378567 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.378538 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.384877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.384847 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm"] Oct 30 20:55:18.387032 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.387006 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:55:18.462837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.462775 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" podStartSLOduration=5.462753524 podStartE2EDuration="5.462753524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:18.453904652 +0000 UTC m=+931.812617391" watchObservedRunningTime="2026-10-30 20:55:18.462753524 +0000 UTC m=+931.821466271" Oct 30 20:55:18.544217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.536648 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.544217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.536794 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8d4\" (UniqueName: \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.544217 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.536913 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.646724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.646667 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.646724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.646739 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8d4\" (UniqueName: \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.647065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.646799 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.647864 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.647760 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.650120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.650093 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.661608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.661567 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8d4\" (UniqueName: \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\") pod \"maas-api-revoke-sub-782d3e08-nkhkr\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:18.725016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:18.724941 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:19.132668 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:19.132597 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859b27c6_f873_4228_9230_3277a05931da.slice/crio-66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7 WatchSource:0}: Error finding container 66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7: Status 404 returned error can't find the container with id 66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7 Oct 30 20:55:19.134621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.134594 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr"] Oct 30 20:55:19.388395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.388364 5855 generic.go:356] "Generic (PLEG): container finished" podID="006919cd-370a-4fef-9ffa-ca267d4c54c4" containerID="5285df1a7b6da00aa8f0e1df3b26b9e59d58270f1eec782073dc4787f50db398" exitCode=0 Oct 30 20:55:19.388543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.388481 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" event={"ID":"006919cd-370a-4fef-9ffa-ca267d4c54c4","Type":"ContainerDied","Data":"5285df1a7b6da00aa8f0e1df3b26b9e59d58270f1eec782073dc4787f50db398"} Oct 30 20:55:19.390601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.390570 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" event={"ID":"859b27c6-f873-4228-9230-3277a05931da","Type":"ContainerStarted","Data":"66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7"} Oct 30 20:55:19.392524 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.392499 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" event={"ID":"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f","Type":"ContainerStarted","Data":"246657fb1142b9ffb548cb4cf58ef48612665979a34cfa267d25dcb1c08bfea2"} Oct 30 20:55:19.392687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.392542 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" event={"ID":"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f","Type":"ContainerStarted","Data":"7620e30bd1e80cc7b2542330fd1d0c1325653091e30a05674018e273ff2f45c3"} Oct 30 20:55:19.436371 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:19.436302 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" podStartSLOduration=4.436285074 podStartE2EDuration="4.436285074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:19.433363133 +0000 UTC m=+932.792075906" watchObservedRunningTime="2026-10-30 20:55:19.436285074 +0000 UTC m=+932.794997820" Oct 30 20:55:20.453853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.453607 5855 generic.go:356] "Generic (PLEG): container finished" podID="859b27c6-f873-4228-9230-3277a05931da" containerID="196e8d08230bc6a83f48872052028ae4271ac65aef75419f84465a7ac952e704" exitCode=0 Oct 30 20:55:20.454483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.454449 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" event={"ID":"859b27c6-f873-4228-9230-3277a05931da","Type":"ContainerDied","Data":"196e8d08230bc6a83f48872052028ae4271ac65aef75419f84465a7ac952e704"} Oct 30 20:55:20.457827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.457794 5855 generic.go:356] "Generic (PLEG): container finished" podID="c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" containerID="246657fb1142b9ffb548cb4cf58ef48612665979a34cfa267d25dcb1c08bfea2" exitCode=0 Oct 30 20:55:20.458170 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.458008 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" event={"ID":"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f","Type":"ContainerDied","Data":"246657fb1142b9ffb548cb4cf58ef48612665979a34cfa267d25dcb1c08bfea2"} Oct 30 20:55:20.751895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.751866 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:20.838618 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.838585 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\") pod \"006919cd-370a-4fef-9ffa-ca267d4c54c4\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " Oct 30 20:55:20.838881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.838862 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\" (UniqueName: \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\") pod \"006919cd-370a-4fef-9ffa-ca267d4c54c4\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " Oct 30 20:55:20.839006 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.838991 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\") pod \"006919cd-370a-4fef-9ffa-ca267d4c54c4\" (UID: \"006919cd-370a-4fef-9ffa-ca267d4c54c4\") " Oct 30 20:55:20.839430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.839350 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca" pod "006919cd-370a-4fef-9ffa-ca267d4c54c4" (UID: "006919cd-370a-4fef-9ffa-ca267d4c54c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:20.839593 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.839576 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:20.840993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.840965 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca" pod "006919cd-370a-4fef-9ffa-ca267d4c54c4" (UID: "006919cd-370a-4fef-9ffa-ca267d4c54c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:20.841082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.841023 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l" pod "006919cd-370a-4fef-9ffa-ca267d4c54c4" (UID: "006919cd-370a-4fef-9ffa-ca267d4c54c4"). InnerVolumeSpecName "kube-api-access-r226l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:20.941851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.941819 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r226l\" (UniqueName: \"kubernetes.io/projected/006919cd-370a-4fef-9ffa-ca267d4c54c4-kube-api-access-r226l\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:20.941851 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:20.941847 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006919cd-370a-4fef-9ffa-ca267d4c54c4-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.469991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.469959 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" Oct 30 20:55:21.470604 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.470001 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7" event={"ID":"006919cd-370a-4fef-9ffa-ca267d4c54c4","Type":"ContainerDied","Data":"fe83f8039e56c33a394a3e1cd726b8a471e8a882aa69cc8c56ecb7f6c9d4f974"} Oct 30 20:55:21.470604 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.470057 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe83f8039e56c33a394a3e1cd726b8a471e8a882aa69cc8c56ecb7f6c9d4f974" Oct 30 20:55:21.651326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.651290 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:21.673103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.673056 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:21.759187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759157 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\" (UniqueName: \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\") pod \"859b27c6-f873-4228-9230-3277a05931da\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " Oct 30 20:55:21.759341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759214 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\") pod \"859b27c6-f873-4228-9230-3277a05931da\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " Oct 30 20:55:21.759341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759327 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\") pod \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " Oct 30 20:55:21.759433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759395 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\") pod \"859b27c6-f873-4228-9230-3277a05931da\" (UID: \"859b27c6-f873-4228-9230-3277a05931da\") " Oct 30 20:55:21.759510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759495 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\" (UniqueName: \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\") pod \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " Oct 30 20:55:21.759587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759555 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\") pod \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\" (UID: \"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f\") " Oct 30 20:55:21.759681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759593 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca" pod "859b27c6-f873-4228-9230-3277a05931da" (UID: "859b27c6-f873-4228-9230-3277a05931da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:21.759999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.759971 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca" pod "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" (UID: "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:21.760343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.760317 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859b27c6-f873-4228-9230-3277a05931da-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.760449 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.760346 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.762543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.762519 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca" pod "859b27c6-f873-4228-9230-3277a05931da" (UID: "859b27c6-f873-4228-9230-3277a05931da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:21.762803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.762781 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca" pod "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" (UID: "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:21.762905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.762814 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4" pod "859b27c6-f873-4228-9230-3277a05931da" (UID: "859b27c6-f873-4228-9230-3277a05931da"). InnerVolumeSpecName "kube-api-access-4d8d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:21.762977 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.762962 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k" pod "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" (UID: "c0fa74c9-28cf-4a03-9e37-8f0894a43a8f"). InnerVolumeSpecName "kube-api-access-2g58k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:21.863855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.863818 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8d4\" (UniqueName: \"kubernetes.io/projected/859b27c6-f873-4228-9230-3277a05931da-kube-api-access-4d8d4\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.863855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.863855 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859b27c6-f873-4228-9230-3277a05931da-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.864096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.863882 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g58k\" (UniqueName: \"kubernetes.io/projected/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-kube-api-access-2g58k\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:21.864096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:21.863897 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:22.324307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.324263 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr"] Oct 30 20:55:22.325236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325216 5855 cpu_manager.go:414] "containerMap: removing container" podUID="859b27c6-f873-4228-9230-3277a05931da" containerName="revoke-keys" Oct 30 20:55:22.325291 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325239 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859b27c6-f873-4228-9230-3277a05931da" containerName="revoke-keys" Oct 30 20:55:22.325326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325292 5855 cpu_manager.go:414] "containerMap: removing container" podUID="006919cd-370a-4fef-9ffa-ca267d4c54c4" containerName="revoke-keys" Oct 30 20:55:22.325326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325300 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006919cd-370a-4fef-9ffa-ca267d4c54c4" containerName="revoke-keys" Oct 30 20:55:22.325326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325310 5855 cpu_manager.go:414] "containerMap: removing container" podUID="c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" containerName="revoke-keys" Oct 30 20:55:22.325326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325321 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" containerName="revoke-keys" Oct 30 20:55:22.325470 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325460 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="006919cd-370a-4fef-9ffa-ca267d4c54c4" containerName="revoke-keys" Oct 30 20:55:22.325504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325474 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" containerName="revoke-keys" Oct 30 20:55:22.325504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.325489 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="859b27c6-f873-4228-9230-3277a05931da" containerName="revoke-keys" Oct 30 20:55:22.330506 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.330479 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.334504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.334471 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr"] Oct 30 20:55:22.482065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.481926 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" Oct 30 20:55:22.482065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.481954 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr" event={"ID":"859b27c6-f873-4228-9230-3277a05931da","Type":"ContainerDied","Data":"66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7"} Oct 30 20:55:22.482065 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.482014 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e7bfb9c191056d466c324f23b6c5eda411e848a29ae269bebd38270da9ccf7" Oct 30 20:55:22.482630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.482194 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.482630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.482305 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.482770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.482698 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmhb\" (UniqueName: \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.486671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.486620 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" event={"ID":"c0fa74c9-28cf-4a03-9e37-8f0894a43a8f","Type":"ContainerDied","Data":"7620e30bd1e80cc7b2542330fd1d0c1325653091e30a05674018e273ff2f45c3"} Oct 30 20:55:22.486830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.486702 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7620e30bd1e80cc7b2542330fd1d0c1325653091e30a05674018e273ff2f45c3" Oct 30 20:55:22.486830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.486771 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm" Oct 30 20:55:22.584881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.584787 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.585044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.584883 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.585044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.584920 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmhb\" (UniqueName: \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.585587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.585567 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.587551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.587527 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.599285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.599260 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmhb\" (UniqueName: \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\") pod \"maas-api-revoke-sub-eefa7b1c-xjjjr\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.652499 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.652460 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:22.964482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:22.961465 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr"] Oct 30 20:55:22.964482 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:22.963537 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179781f2_fc14_4b7e_b76c_5c8427e1ced7.slice/crio-c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97 WatchSource:0}: Error finding container c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97: Status 404 returned error can't find the container with id c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97 Oct 30 20:55:23.498859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:23.498815 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" event={"ID":"179781f2-fc14-4b7e-b76c-5c8427e1ced7","Type":"ContainerStarted","Data":"632edbd275b026c7240e4c28bc20acfb79081501a9303acf07c856624834cb38"} Oct 30 20:55:23.499358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:23.498889 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" event={"ID":"179781f2-fc14-4b7e-b76c-5c8427e1ced7","Type":"ContainerStarted","Data":"c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97"} Oct 30 20:55:23.522987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:23.522918 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" podStartSLOduration=1.522895987 podStartE2EDuration="1.522895987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:23.518594064 +0000 UTC m=+936.877306812" watchObservedRunningTime="2026-10-30 20:55:23.522895987 +0000 UTC m=+936.881608740" Oct 30 20:55:24.372875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.372838 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6"] Oct 30 20:55:24.377490 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.377464 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.385003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.384969 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6"] Oct 30 20:55:24.468719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.466075 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.468719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.466298 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlhq\" (UniqueName: \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.468719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.466440 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.511759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.511718 5855 generic.go:356] "Generic (PLEG): container finished" podID="179781f2-fc14-4b7e-b76c-5c8427e1ced7" containerID="632edbd275b026c7240e4c28bc20acfb79081501a9303acf07c856624834cb38" exitCode=0 Oct 30 20:55:24.512182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.511974 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" event={"ID":"179781f2-fc14-4b7e-b76c-5c8427e1ced7","Type":"ContainerDied","Data":"632edbd275b026c7240e4c28bc20acfb79081501a9303acf07c856624834cb38"} Oct 30 20:55:24.568207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.568174 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlhq\" (UniqueName: \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.568398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.568306 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.568467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.568452 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.569128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.569097 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.571073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.571050 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.582017 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.581985 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlhq\" (UniqueName: \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\") pod \"maas-api-revoke-sub-22fd3b57-5fch6\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:24.713735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:24.712030 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:25.091612 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:25.091572 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820b0b26_5bdd_435c_abb1_cf7af82b181c.slice/crio-7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9 WatchSource:0}: Error finding container 7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9: Status 404 returned error can't find the container with id 7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9 Oct 30 20:55:25.092052 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.092028 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6"] Oct 30 20:55:25.523096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.523062 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" event={"ID":"820b0b26-5bdd-435c-abb1-cf7af82b181c","Type":"ContainerStarted","Data":"b572ec05af989619c2dcce8b0e326b4383cd8f7ea35e356ad315cd61678e002e"} Oct 30 20:55:25.523562 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.523121 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" event={"ID":"820b0b26-5bdd-435c-abb1-cf7af82b181c","Type":"ContainerStarted","Data":"7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9"} Oct 30 20:55:25.720599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.720572 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:25.743068 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.742954 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" podStartSLOduration=1.742934588 podStartE2EDuration="1.742934588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:25.54828411 +0000 UTC m=+938.906996846" watchObservedRunningTime="2026-10-30 20:55:25.742934588 +0000 UTC m=+939.101647335" Oct 30 20:55:25.784308 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.784269 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt"] Oct 30 20:55:25.785251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.785223 5855 cpu_manager.go:414] "containerMap: removing container" podUID="179781f2-fc14-4b7e-b76c-5c8427e1ced7" containerName="revoke-keys" Oct 30 20:55:25.785251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.785250 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179781f2-fc14-4b7e-b76c-5c8427e1ced7" containerName="revoke-keys" Oct 30 20:55:25.785464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.785431 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="179781f2-fc14-4b7e-b76c-5c8427e1ced7" containerName="revoke-keys" Oct 30 20:55:25.790077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.790053 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.804331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.804295 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt"] Oct 30 20:55:25.837201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837174 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\" (UniqueName: \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\") pod \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " Oct 30 20:55:25.837368 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837348 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\") pod \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " Oct 30 20:55:25.837460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837447 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\") pod \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\" (UID: \"179781f2-fc14-4b7e-b76c-5c8427e1ced7\") " Oct 30 20:55:25.837740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837720 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.837862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837748 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.837862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837790 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca" pod "179781f2-fc14-4b7e-b76c-5c8427e1ced7" (UID: "179781f2-fc14-4b7e-b76c-5c8427e1ced7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:25.837946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.837913 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9t9\" (UniqueName: \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.838206 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.838187 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:25.839419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.839396 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb" pod "179781f2-fc14-4b7e-b76c-5c8427e1ced7" (UID: "179781f2-fc14-4b7e-b76c-5c8427e1ced7"). InnerVolumeSpecName "kube-api-access-2fmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:25.839621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.839602 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca" pod "179781f2-fc14-4b7e-b76c-5c8427e1ced7" (UID: "179781f2-fc14-4b7e-b76c-5c8427e1ced7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:25.939156 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939127 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.939156 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939160 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.939411 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939213 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9t9\" (UniqueName: \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.939542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939523 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179781f2-fc14-4b7e-b76c-5c8427e1ced7-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:25.939603 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939551 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmhb\" (UniqueName: \"kubernetes.io/projected/179781f2-fc14-4b7e-b76c-5c8427e1ced7-kube-api-access-2fmhb\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:25.939929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.939896 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.941876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.941854 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:25.952256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:25.952231 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9t9\" (UniqueName: \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\") pod \"maas-api-revoke-sub-77cee663-xkmwt\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:26.118281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.118247 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:26.475173 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:26.475135 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b953826_67de_4def_827f_23e6d6e3d414.slice/crio-b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4 WatchSource:0}: Error finding container b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4: Status 404 returned error can't find the container with id b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4 Oct 30 20:55:26.475395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.475377 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt"] Oct 30 20:55:26.537593 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.537554 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" Oct 30 20:55:26.538082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.537743 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr" event={"ID":"179781f2-fc14-4b7e-b76c-5c8427e1ced7","Type":"ContainerDied","Data":"c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97"} Oct 30 20:55:26.538082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.537797 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55164d4a7ed92a4ff94c624ed4906bd1b8f20081ec1d23496f81767494e6f97" Oct 30 20:55:26.540951 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.540919 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" event={"ID":"4b953826-67de-4def-827f-23e6d6e3d414","Type":"ContainerStarted","Data":"b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4"} Oct 30 20:55:26.543005 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.542976 5855 generic.go:356] "Generic (PLEG): container finished" podID="820b0b26-5bdd-435c-abb1-cf7af82b181c" containerID="b572ec05af989619c2dcce8b0e326b4383cd8f7ea35e356ad315cd61678e002e" exitCode=0 Oct 30 20:55:26.543145 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:26.543085 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" event={"ID":"820b0b26-5bdd-435c-abb1-cf7af82b181c","Type":"ContainerDied","Data":"b572ec05af989619c2dcce8b0e326b4383cd8f7ea35e356ad315cd61678e002e"} Oct 30 20:55:27.555346 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.555302 5855 generic.go:356] "Generic (PLEG): container finished" podID="4b953826-67de-4def-827f-23e6d6e3d414" containerID="920890d07b8fe504ff0bb5a4181065f83c8d6ea428189870cb73f02e0dea374d" exitCode=0 Oct 30 20:55:27.555840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.555395 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" event={"ID":"4b953826-67de-4def-827f-23e6d6e3d414","Type":"ContainerDied","Data":"920890d07b8fe504ff0bb5a4181065f83c8d6ea428189870cb73f02e0dea374d"} Oct 30 20:55:27.903753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.903649 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:27.944299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.944259 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\") pod \"820b0b26-5bdd-435c-abb1-cf7af82b181c\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " Oct 30 20:55:27.944497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.944396 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\") pod \"820b0b26-5bdd-435c-abb1-cf7af82b181c\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " Oct 30 20:55:27.944572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.944550 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\" (UniqueName: \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\") pod \"820b0b26-5bdd-435c-abb1-cf7af82b181c\" (UID: \"820b0b26-5bdd-435c-abb1-cf7af82b181c\") " Oct 30 20:55:27.944722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.944692 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca" pod "820b0b26-5bdd-435c-abb1-cf7af82b181c" (UID: "820b0b26-5bdd-435c-abb1-cf7af82b181c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:27.945800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.945771 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:27.946777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.946744 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca" pod "820b0b26-5bdd-435c-abb1-cf7af82b181c" (UID: "820b0b26-5bdd-435c-abb1-cf7af82b181c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:27.948855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:27.948820 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq" pod "820b0b26-5bdd-435c-abb1-cf7af82b181c" (UID: "820b0b26-5bdd-435c-abb1-cf7af82b181c"). InnerVolumeSpecName "kube-api-access-pzlhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:28.047235 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.047209 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/820b0b26-5bdd-435c-abb1-cf7af82b181c-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:28.047415 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.047239 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlhq\" (UniqueName: \"kubernetes.io/projected/820b0b26-5bdd-435c-abb1-cf7af82b181c-kube-api-access-pzlhq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:28.567371 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.567341 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" Oct 30 20:55:28.567929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.567874 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6" event={"ID":"820b0b26-5bdd-435c-abb1-cf7af82b181c","Type":"ContainerDied","Data":"7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9"} Oct 30 20:55:28.567929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.567926 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4560bd119ea85a007e33b3ca20bdf73b429a89d68acd83f767d0a4274117b9" Oct 30 20:55:28.772889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.772856 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:28.864203 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.864107 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\" (UniqueName: \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\") pod \"4b953826-67de-4def-827f-23e6d6e3d414\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " Oct 30 20:55:28.864203 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.864188 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\") pod \"4b953826-67de-4def-827f-23e6d6e3d414\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " Oct 30 20:55:28.864460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.864384 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\") pod \"4b953826-67de-4def-827f-23e6d6e3d414\" (UID: \"4b953826-67de-4def-827f-23e6d6e3d414\") " Oct 30 20:55:28.864808 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.864780 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca" pod "4b953826-67de-4def-827f-23e6d6e3d414" (UID: "4b953826-67de-4def-827f-23e6d6e3d414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:28.865109 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.865090 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:28.866469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.866442 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca" pod "4b953826-67de-4def-827f-23e6d6e3d414" (UID: "4b953826-67de-4def-827f-23e6d6e3d414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:28.866612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.866592 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9" pod "4b953826-67de-4def-827f-23e6d6e3d414" (UID: "4b953826-67de-4def-827f-23e6d6e3d414"). InnerVolumeSpecName "kube-api-access-7w9t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:28.891553 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.891518 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn"] Oct 30 20:55:28.892269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892248 5855 cpu_manager.go:414] "containerMap: removing container" podUID="820b0b26-5bdd-435c-abb1-cf7af82b181c" containerName="revoke-keys" Oct 30 20:55:28.892376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892272 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820b0b26-5bdd-435c-abb1-cf7af82b181c" containerName="revoke-keys" Oct 30 20:55:28.892376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892331 5855 cpu_manager.go:414] "containerMap: removing container" podUID="4b953826-67de-4def-827f-23e6d6e3d414" containerName="revoke-keys" Oct 30 20:55:28.892376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892338 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b953826-67de-4def-827f-23e6d6e3d414" containerName="revoke-keys" Oct 30 20:55:28.892535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892482 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="820b0b26-5bdd-435c-abb1-cf7af82b181c" containerName="revoke-keys" Oct 30 20:55:28.892535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.892504 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b953826-67de-4def-827f-23e6d6e3d414" containerName="revoke-keys" Oct 30 20:55:28.898372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.897242 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:28.909408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.909363 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn"] Oct 30 20:55:28.966020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.965981 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ph7\" (UniqueName: \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:28.966389 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.966366 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:28.966606 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.966590 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:28.966766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.966751 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9t9\" (UniqueName: \"kubernetes.io/projected/4b953826-67de-4def-827f-23e6d6e3d414-kube-api-access-7w9t9\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:28.966834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:28.966773 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b953826-67de-4def-827f-23e6d6e3d414-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:29.068168 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.068129 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.068349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.068207 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ph7\" (UniqueName: \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.068435 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.068361 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.068934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.068910 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.071053 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.071029 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.082015 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.081973 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ph7\" (UniqueName: \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\") pod \"maas-api-revoke-sub-a067a0da-kzccn\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.233732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.232959 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:29.577054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.577020 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" Oct 30 20:55:29.577054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.577041 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt" event={"ID":"4b953826-67de-4def-827f-23e6d6e3d414","Type":"ContainerDied","Data":"b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4"} Oct 30 20:55:29.577571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.577089 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72a024e2ed6197d9294a1a4cad9e3cb172e82566acb7330188ce48ee18bd4e4" Oct 30 20:55:29.617319 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:29.617281 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f22af8e_6941_4033_af47_157af709f986.slice/crio-bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66 WatchSource:0}: Error finding container bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66: Status 404 returned error can't find the container with id bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66 Oct 30 20:55:29.618461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:29.618431 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn"] Oct 30 20:55:30.128188 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.128146 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p"] Oct 30 20:55:30.174706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.173756 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p"] Oct 30 20:55:30.174706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.173983 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.314035 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.313899 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.314035 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.314034 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.314328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.314183 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9cq\" (UniqueName: \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.416279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.416230 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9cq\" (UniqueName: \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.416546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.416529 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.416687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.416644 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.417274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.417247 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.419248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.419224 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.429299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.429268 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9cq\" (UniqueName: \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\") pod \"maas-api-revoke-sub-cac26579-hq62p\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.534748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.534710 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:30.591142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.591105 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" event={"ID":"2f22af8e-6941-4033-af47-157af709f986","Type":"ContainerStarted","Data":"f7cdff769eee65506a9c5f7aa41d60a15365ded954d3ee08c708630f1f5b7f91"} Oct 30 20:55:30.591563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.591179 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" event={"ID":"2f22af8e-6941-4033-af47-157af709f986","Type":"ContainerStarted","Data":"bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66"} Oct 30 20:55:30.629859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.629783 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" podStartSLOduration=2.629763844 podStartE2EDuration="2.629763844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:30.626088534 +0000 UTC m=+943.984801280" watchObservedRunningTime="2026-10-30 20:55:30.629763844 +0000 UTC m=+943.988476592" Oct 30 20:55:30.860484 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:30.860428 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5278b0_2192_40b2_b320_a87816377803.slice/crio-7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609 WatchSource:0}: Error finding container 7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609: Status 404 returned error can't find the container with id 7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609 Oct 30 20:55:30.880740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:30.877361 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p"] Oct 30 20:55:30.934193 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:55:30.934157 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:55:31.604998 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:31.604962 5855 generic.go:356] "Generic (PLEG): container finished" podID="2f22af8e-6941-4033-af47-157af709f986" containerID="f7cdff769eee65506a9c5f7aa41d60a15365ded954d3ee08c708630f1f5b7f91" exitCode=0 Oct 30 20:55:31.605479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:31.605116 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" event={"ID":"2f22af8e-6941-4033-af47-157af709f986","Type":"ContainerDied","Data":"f7cdff769eee65506a9c5f7aa41d60a15365ded954d3ee08c708630f1f5b7f91"} Oct 30 20:55:31.607609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:31.607582 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" event={"ID":"bb5278b0-2192-40b2-b320-a87816377803","Type":"ContainerStarted","Data":"c1ea1d4b436f89ac0976fedcf355cc497092a4cee6bd11e9b78a1364d44f0742"} Oct 30 20:55:31.607838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:31.607819 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" event={"ID":"bb5278b0-2192-40b2-b320-a87816377803","Type":"ContainerStarted","Data":"7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609"} Oct 30 20:55:31.647004 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:31.646948 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" podStartSLOduration=1.646932756 podStartE2EDuration="1.646932756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:31.644550938 +0000 UTC m=+945.003263686" watchObservedRunningTime="2026-10-30 20:55:31.646932756 +0000 UTC m=+945.005645536" Oct 30 20:55:32.273215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.273132 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 20:55:32.278051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.278026 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.294478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.294441 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 20:55:32.380561 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.380519 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.380777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.380610 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.380819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.380770 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.380819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.380808 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.380983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.380965 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.381043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.381029 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmz8f\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.381205 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.381184 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.381253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.381242 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.381349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.381339 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.482906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.482868 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmz8f\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483001 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483048 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483096 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483124 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483165 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483222 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483243 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483334 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483558 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483531 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483721 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.483864 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.483836 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.484022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.484003 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.484097 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.484041 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.486215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.486186 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.486513 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.486494 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.496357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.496325 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmz8f\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.496559 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.496534 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.617231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.617198 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:32.618085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.618059 5855 generic.go:356] "Generic (PLEG): container finished" podID="bb5278b0-2192-40b2-b320-a87816377803" containerID="c1ea1d4b436f89ac0976fedcf355cc497092a4cee6bd11e9b78a1364d44f0742" exitCode=0 Oct 30 20:55:32.618175 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.618147 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" event={"ID":"bb5278b0-2192-40b2-b320-a87816377803","Type":"ContainerDied","Data":"c1ea1d4b436f89ac0976fedcf355cc497092a4cee6bd11e9b78a1364d44f0742"} Oct 30 20:55:32.858629 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.858240 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:32.993461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.993364 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\" (UniqueName: \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\") pod \"2f22af8e-6941-4033-af47-157af709f986\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " Oct 30 20:55:32.993983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.993952 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\") pod \"2f22af8e-6941-4033-af47-157af709f986\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " Oct 30 20:55:32.994197 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:32.994161 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\") pod \"2f22af8e-6941-4033-af47-157af709f986\" (UID: \"2f22af8e-6941-4033-af47-157af709f986\") " Oct 30 20:55:33.007210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.007138 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca" pod "2f22af8e-6941-4033-af47-157af709f986" (UID: "2f22af8e-6941-4033-af47-157af709f986"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:33.016630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.016573 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7" pod "2f22af8e-6941-4033-af47-157af709f986" (UID: "2f22af8e-6941-4033-af47-157af709f986"). InnerVolumeSpecName "kube-api-access-v5ph7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:33.029794 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.025582 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ph7\" (UniqueName: \"kubernetes.io/projected/2f22af8e-6941-4033-af47-157af709f986-kube-api-access-v5ph7\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:33.029794 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.025623 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f22af8e-6941-4033-af47-157af709f986-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:33.051972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.051930 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca" pod "2f22af8e-6941-4033-af47-157af709f986" (UID: "2f22af8e-6941-4033-af47-157af709f986"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:33.070695 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.065999 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 20:55:33.077141 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.077081 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:33.077463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.077421 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:33.077626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.077487 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:33.143693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.142362 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f22af8e-6941-4033-af47-157af709f986-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:33.630348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.630320 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" Oct 30 20:55:33.630838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.630358 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn" event={"ID":"2f22af8e-6941-4033-af47-157af709f986","Type":"ContainerDied","Data":"bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66"} Oct 30 20:55:33.630838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.630414 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1db04c3613d1abdbf5d38801a968533fbd5dc68f5e185b7d6a074485271a66" Oct 30 20:55:33.632726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.632695 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" event={"ID":"18e90219-2c2a-4393-b7a1-dbbd57fc0e32","Type":"ContainerStarted","Data":"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61"} Oct 30 20:55:33.632883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.632752 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" event={"ID":"18e90219-2c2a-4393-b7a1-dbbd57fc0e32","Type":"ContainerStarted","Data":"ad799c8c9fa2ed03869e0a9321ff05ca07b391e96a2ab0cd69d699fb2b0b8bea"} Oct 30 20:55:33.664931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.664874 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podStartSLOduration=1.66485733 podStartE2EDuration="1.66485733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:33.6600967 +0000 UTC m=+947.018809447" watchObservedRunningTime="2026-10-30 20:55:33.66485733 +0000 UTC m=+947.023570076" Oct 30 20:55:33.968060 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:33.968034 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:34.084349 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.084299 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\") pod \"bb5278b0-2192-40b2-b320-a87816377803\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " Oct 30 20:55:34.084834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.084810 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\") pod \"bb5278b0-2192-40b2-b320-a87816377803\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " Oct 30 20:55:34.085050 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.085034 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\" (UniqueName: \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\") pod \"bb5278b0-2192-40b2-b320-a87816377803\" (UID: \"bb5278b0-2192-40b2-b320-a87816377803\") " Oct 30 20:55:34.092119 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.092076 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca" pod "bb5278b0-2192-40b2-b320-a87816377803" (UID: "bb5278b0-2192-40b2-b320-a87816377803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:34.111624 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.111582 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca" pod "bb5278b0-2192-40b2-b320-a87816377803" (UID: "bb5278b0-2192-40b2-b320-a87816377803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:34.111882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.111826 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq" pod "bb5278b0-2192-40b2-b320-a87816377803" (UID: "bb5278b0-2192-40b2-b320-a87816377803"). InnerVolumeSpecName "kube-api-access-nv9cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:34.197743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.197050 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5278b0-2192-40b2-b320-a87816377803-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:34.197743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.197102 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5278b0-2192-40b2-b320-a87816377803-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:34.197743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.197118 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9cq\" (UniqueName: \"kubernetes.io/projected/bb5278b0-2192-40b2-b320-a87816377803-kube-api-access-nv9cq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:34.631882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.631852 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:34.632772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.632649 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:34.633033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.633003 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:34.690535 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.690498 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" event={"ID":"bb5278b0-2192-40b2-b320-a87816377803","Type":"ContainerDied","Data":"7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609"} Oct 30 20:55:34.690908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.690885 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b16390ee895dc6a4d7a78da855d13b4870104b1abad965774120da988f0d609" Oct 30 20:55:34.691064 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.690809 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p" Oct 30 20:55:34.863402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.863369 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n"] Oct 30 20:55:34.864503 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864478 5855 cpu_manager.go:414] "containerMap: removing container" podUID="bb5278b0-2192-40b2-b320-a87816377803" containerName="revoke-keys" Oct 30 20:55:34.864604 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864504 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5278b0-2192-40b2-b320-a87816377803" containerName="revoke-keys" Oct 30 20:55:34.864604 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864587 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2f22af8e-6941-4033-af47-157af709f986" containerName="revoke-keys" Oct 30 20:55:34.864755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864597 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f22af8e-6941-4033-af47-157af709f986" containerName="revoke-keys" Oct 30 20:55:34.864876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864861 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5278b0-2192-40b2-b320-a87816377803" containerName="revoke-keys" Oct 30 20:55:34.864939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.864893 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f22af8e-6941-4033-af47-157af709f986" containerName="revoke-keys" Oct 30 20:55:34.870003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.869978 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n"] Oct 30 20:55:34.870110 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.870038 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:34.909758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.906604 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:55:34.909758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.906734 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:55:34.909758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.906822 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:55:34.909758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:34.906870 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:35.054090 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.054053 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.054339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.054309 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm9p\" (UniqueName: \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.055253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.055233 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.156249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.156200 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.156466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.156414 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.156553 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.156537 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm9p\" (UniqueName: \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.157077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.157042 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.159025 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.158993 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.170184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.170092 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm9p\" (UniqueName: \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\") pod \"maas-api-revoke-sub-edba010c-gpw2n\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.226534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.226487 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:35.626267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.619127 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:35.626267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.619204 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:35.648682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:35.642065 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n"] Oct 30 20:55:35.746346 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:35.746274 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d534605_ae0b_48c6_a461_7db789d16547.slice/crio-42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c WatchSource:0}: Error finding container 42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c: Status 404 returned error can't find the container with id 42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c Oct 30 20:55:36.618609 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:36.618573 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:36.618906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:36.618877 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:36.768571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:36.768529 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" event={"ID":"7d534605-ae0b-48c6-a461-7db789d16547","Type":"ContainerStarted","Data":"af7725722fbdd930cd4d60eb1d8f60d6cbe46673069048f2faff8de45a1f5854"} Oct 30 20:55:36.769103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:36.768588 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" event={"ID":"7d534605-ae0b-48c6-a461-7db789d16547","Type":"ContainerStarted","Data":"42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c"} Oct 30 20:55:36.822245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:36.822186 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" podStartSLOduration=2.822167671 podStartE2EDuration="2.822167671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:36.811283152 +0000 UTC m=+950.169995898" watchObservedRunningTime="2026-10-30 20:55:36.822167671 +0000 UTC m=+950.180880418" Oct 30 20:55:37.622113 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:37.622075 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:37.622457 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:37.622426 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:37.801715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:37.798501 5855 generic.go:356] "Generic (PLEG): container finished" podID="7d534605-ae0b-48c6-a461-7db789d16547" containerID="af7725722fbdd930cd4d60eb1d8f60d6cbe46673069048f2faff8de45a1f5854" exitCode=0 Oct 30 20:55:37.801715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:37.798839 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" event={"ID":"7d534605-ae0b-48c6-a461-7db789d16547","Type":"ContainerDied","Data":"af7725722fbdd930cd4d60eb1d8f60d6cbe46673069048f2faff8de45a1f5854"} Oct 30 20:55:38.021161 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.019609 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl"] Oct 30 20:55:38.027576 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.027506 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.031886 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.031430 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl"] Oct 30 20:55:38.104688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.083137 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpgt\" (UniqueName: \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.107011 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.106974 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.107329 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.107310 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.210187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.210149 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpgt\" (UniqueName: \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.210408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.210301 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.210408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.210345 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.211230 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.211203 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.214210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.213418 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.224324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.224296 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpgt\" (UniqueName: \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\") pod \"maas-api-revoke-sub-2d1f5014-g9cxl\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.359599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.359568 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:38.754740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.750984 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:38.754740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:38.751089 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:39.525741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.512811 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.600744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.530745 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:39.706683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.692484 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:39.706683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.692569 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:39.734680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.707804 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.734680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.731237 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.734680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.732775 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.751849 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.751810 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:39.756133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.756099 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:39.783377 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:55:39.770783 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05177ebd_27f4_479a_b459_816cb238dc4b.slice/crio-91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260 WatchSource:0}: Error finding container 91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260: Status 404 returned error can't find the container with id 91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260 Oct 30 20:55:39.783377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.776228 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:39.793800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.793760 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Oct 30 20:55:39.860416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.860370 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.959117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.905943 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:39.959117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:39.907244 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:40.140684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.121558 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 20:55:40.162685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.159055 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:40.170782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.170733 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl"] Oct 30 20:55:40.171209 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.171178 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:40.540681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.536809 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:40.540681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.540007 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:55:40.556342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.556240 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:40.558442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.558414 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:55:40.633714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.615519 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" event={"ID":"05177ebd-27f4-479a-b459-816cb238dc4b","Type":"ContainerStarted","Data":"91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260"} Oct 30 20:55:40.637288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.637214 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:40.637495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.637300 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:40.835116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.724763 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:55:40.835116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.808690 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:40.858147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.855922 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:40.858147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.856449 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:40.858147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.856838 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:40.867786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.867750 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:41.302151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.870328 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:41.440482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.936831 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.441450 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.945775 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.465203 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:40.957776 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.501684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.282603 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.282670 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.549334 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.613674 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.687300 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.687987 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.695207 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:41.718764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.716797 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:41.772927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.443636 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.809991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.809926 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:41.863377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.863320 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.863792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.863760 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.864049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.864025 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:41.871883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.871811 5855 patch_prober.go:29] interesting pod/metrics-server-656d8b8c49-pmmgw container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:41.899681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.872103 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-656d8b8c49-pmmgw" podUID="3ac2e542-da70-4445-a126-d02866a7e2f9" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.12:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.949879 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.983021 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999135 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999523 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999603 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999644 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999701 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999744 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999779 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:55:42.027781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:41.999812 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:55:42.277686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.161351 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.463002 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.452484 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:42.469839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.469788 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:42.522708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.507756 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.555751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.523167 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:42.555751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.523235 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.555751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.526578 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.637009 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.636963 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:42.670803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.670764 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:42.727703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.721749 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.968925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.968826 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:42.970821 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.970782 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.978890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.978841 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:42.988684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.981496 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:42.992990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.992946 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:43.011468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:42.994301 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.039909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.013959 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.109531 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.106751 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.278542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.278449 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.288199 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.288163 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:43.365467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.365422 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:55:43.489912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.469954 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\") pod \"7d534605-ae0b-48c6-a461-7db789d16547\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " Oct 30 20:55:43.489912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.470016 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\") pod \"7d534605-ae0b-48c6-a461-7db789d16547\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " Oct 30 20:55:43.489912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.470121 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\" (UniqueName: \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\") pod \"7d534605-ae0b-48c6-a461-7db789d16547\" (UID: \"7d534605-ae0b-48c6-a461-7db789d16547\") " Oct 30 20:55:43.581001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.571518 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca" pod "7d534605-ae0b-48c6-a461-7db789d16547" (UID: "7d534605-ae0b-48c6-a461-7db789d16547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:43.581001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.572290 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.581001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.572568 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:43.581001 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.572799 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.600311 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.601052 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.640795 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.642842 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.642893 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.676693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.649887 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca" pod "7d534605-ae0b-48c6-a461-7db789d16547" (UID: "7d534605-ae0b-48c6-a461-7db789d16547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:43.691887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.691753 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:43.691887 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.691818 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:43.700732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.699871 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p" pod "7d534605-ae0b-48c6-a461-7db789d16547" (UID: "7d534605-ae0b-48c6-a461-7db789d16547"). InnerVolumeSpecName "kube-api-access-8nm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:43.738003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.737323 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.740780 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.740746 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" event={"ID":"7d534605-ae0b-48c6-a461-7db789d16547","Type":"ContainerDied","Data":"42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c"} Oct 30 20:55:43.741029 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.741008 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d75c6f757f1e2d1303730716685fd703b1a4a1094d647b4810cf8e817d9e1c" Oct 30 20:55:43.741327 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.741310 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n" Oct 30 20:55:43.808165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.808131 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nm9p\" (UniqueName: \"kubernetes.io/projected/7d534605-ae0b-48c6-a461-7db789d16547-kube-api-access-8nm9p\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:43.832192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.832102 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d534605-ae0b-48c6-a461-7db789d16547-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:43.832484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.811688 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" event={"ID":"05177ebd-27f4-479a-b459-816cb238dc4b","Type":"ContainerStarted","Data":"2eef43a18d601ee6b984b7e69b73a8f2e0ff94e3b26b702a5283383cbd5e66f8"} Oct 30 20:55:43.979893 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.979824 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:43.980612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.980574 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:43.991071 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:43.991027 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.008943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.008890 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:44.009705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.009646 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.017778 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.017735 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.051230 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.051167 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.223502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.223335 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:55:44.381679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.379874 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.619680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.614505 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:44.619680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.614894 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" podUID="f7ada5c68d422440bbf7fd2285220de0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 20:55:44.619680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.614983 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:44.640157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.628353 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:44.640157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.631550 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:44.640157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.632182 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:44.640157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.632222 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:44.769313 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.739762 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.823688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.822830 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Oct 30 20:55:44.973091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.947118 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:44.973091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.947177 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.008762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.985318 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:45.008762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:44.985876 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.009183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.009145 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:45.011405 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.011369 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:45.060835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.026758 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.061582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.061533 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:45.084160 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.083799 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:45.084160 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.083858 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.347684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.334547 5855 generic.go:356] "Generic (PLEG): container finished" podID="05177ebd-27f4-479a-b459-816cb238dc4b" containerID="2eef43a18d601ee6b984b7e69b73a8f2e0ff94e3b26b702a5283383cbd5e66f8" exitCode=0 Oct 30 20:55:45.348987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.348947 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" event={"ID":"05177ebd-27f4-479a-b459-816cb238dc4b","Type":"ContainerDied","Data":"2eef43a18d601ee6b984b7e69b73a8f2e0ff94e3b26b702a5283383cbd5e66f8"} Oct 30 20:55:45.386713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.386649 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.671783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.643791 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.671783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.644466 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.671783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.644927 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:45.671783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.645324 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.742528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.742010 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.765999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.765959 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:45.766291 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.766261 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:45.992402 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.992297 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.993321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:45.993290 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:46.014179 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.014133 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:55:46.172732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.137434 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:46.191145 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.174427 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:46.192128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.174493 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:46.192548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.174514 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:46.192746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.176141 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:46.279937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.262360 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" podStartSLOduration=9.262334649 podStartE2EDuration="9.262334649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:43.906306154 +0000 UTC m=+957.265018906" watchObservedRunningTime="2026-10-30 20:55:46.262334649 +0000 UTC m=+959.621047392" Oct 30 20:55:46.307825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.307779 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:46.388832 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.388778 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:46.625102 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.625060 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:46.625299 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.625122 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:46.646873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.646814 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:46.647525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.647492 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:46.648005 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.647982 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:46.648432 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.648406 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:46.745742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:46.745699 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:47.013214 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.013112 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:47.014366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.014332 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:47.014942 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.014913 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:47.416466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.416420 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:47.448260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.448218 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:55:47.474686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.450716 5855 cpu_manager.go:414] "containerMap: removing container" podUID="7d534605-ae0b-48c6-a461-7db789d16547" containerName="revoke-keys" Oct 30 20:55:47.513683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.490907 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d534605-ae0b-48c6-a461-7db789d16547" containerName="revoke-keys" Oct 30 20:55:47.513683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.435680 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:47.513683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.435223 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:47.513683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.435471 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:55:47.520762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.470912 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:47.523095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.522968 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d534605-ae0b-48c6-a461-7db789d16547" containerName="revoke-keys" Oct 30 20:55:47.904815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.904763 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:47.972740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:47.957963 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded" Oct 30 20:55:48.064738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.013005 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:48.093461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.093364 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.094266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.017162 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.017182 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.017215 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.017248 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.145293 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.017083 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.145949 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.178993 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.179040 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.179081 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:55:48.220684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.182186 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.268685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.251340 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:55:48.294301 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.252782 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:48.475157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.433566 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:48.475561 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.475537 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:48.480186 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.473767 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.611723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.610803 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:48.613137 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.613083 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.613611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.613589 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:48.613995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.613976 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:48.668410 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.668375 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ab941206-openshift-default-dockercfg-4v5jv\"" Oct 30 20:55:48.679739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.677352 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:48.679739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.677411 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:48.696511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.696462 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:48.696850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.696803 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:48.779772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.779730 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780052 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780032 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780150 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7d7x\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780284 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780271 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780381 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780370 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780570 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.780761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.780747 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.781252 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.781232 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.781431 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.781418 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.900520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.884851 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.979534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.979493 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.979936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.979909 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.980136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.980117 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.980424 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.980400 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.980586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.980570 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.980737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.980721 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7d7x\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.980929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.980910 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.981056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.981040 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.981707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.918361 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.985105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.985070 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:48.985387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.948547 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:55:48.986813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:48.986774 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.014525 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.014484 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.015178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.015146 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.039700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.027737 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.049682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.042378 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.050677 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.050616 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7d7x\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.078162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.078121 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\") pod \"e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.104898 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.097863 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.112124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.102764 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:49.112648 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.112613 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.127752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.116966 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:49.152943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.152837 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.153404 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.153372 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:49.154290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.153847 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.180426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.153882 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:49.185389 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.153901 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:49.194715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.192526 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\") pod \"05177ebd-27f4-479a-b459-816cb238dc4b\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " Oct 30 20:55:49.194715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.192768 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\" (UniqueName: \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\") pod \"05177ebd-27f4-479a-b459-816cb238dc4b\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " Oct 30 20:55:49.194715 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.192874 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\") pod \"05177ebd-27f4-479a-b459-816cb238dc4b\" (UID: \"05177ebd-27f4-479a-b459-816cb238dc4b\") " Oct 30 20:55:49.271731 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.271670 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.337686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.281120 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca" pod "05177ebd-27f4-479a-b459-816cb238dc4b" (UID: "05177ebd-27f4-479a-b459-816cb238dc4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:49.337686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.319872 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:49.458727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.444068 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:49.458727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.450772 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca" pod "05177ebd-27f4-479a-b459-816cb238dc4b" (UID: "05177ebd-27f4-479a-b459-816cb238dc4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:49.467174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.467035 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt" pod "05177ebd-27f4-479a-b459-816cb238dc4b" (UID: "05177ebd-27f4-479a-b459-816cb238dc4b"). InnerVolumeSpecName "kube-api-access-snpgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:49.511702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.497148 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.511702 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.497547 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.616693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.611957 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:55:49.637706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.637674 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpgt\" (UniqueName: \"kubernetes.io/projected/05177ebd-27f4-479a-b459-816cb238dc4b-kube-api-access-snpgt\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:49.637985 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.637958 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05177ebd-27f4-479a-b459-816cb238dc4b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:49.640251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.640213 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.645790 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.645747 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:49.646386 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.646354 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.647115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.647087 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:49.748082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.725552 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:49.748082 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.725613 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:49.790874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.790846 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:55:49.862792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.862754 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" event={"ID":"05177ebd-27f4-479a-b459-816cb238dc4b","Type":"ContainerDied","Data":"91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260"} Oct 30 20:55:49.863066 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.863046 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d6d3d1a42fbda995daecbe88c17d658d585196876995286c3f269d84371260" Oct 30 20:55:49.863355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.863339 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl" Oct 30 20:55:49.991748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:49.991531 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.071680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.058082 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.116221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.116180 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:55:50.172224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.172176 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:55:50.209916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.209875 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:55:50.246842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.214734 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:50.275828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.260753 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.319837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.319790 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.448699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.435097 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:55:50.505260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.498874 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.624800 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.624754 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.649681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.646937 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.649681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.647329 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:55:50.649681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.647919 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:50.714911 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.714873 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:55:50.986377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.715193 5855 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:55:50.986377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.763425 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:55:51.004799 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:50.992892 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:55:51.178908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.145579 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.178908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.161194 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:55:51.178908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.161253 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:55:51.178908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.161286 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:55:51.178908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.171604 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:55:51.368872 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.366357 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.373755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.371986 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:55:51.619447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.619348 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.667323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.666427 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:55:51.667323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.666890 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.672328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.672257 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.678873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.678320 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:51.749838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.748770 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:55:51.750043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.749867 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:55:51.770196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.770165 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:55:51.823703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.821300 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:51.823703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.821558 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:51.823703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.821681 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:55:51.866978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.866925 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:55:51.867469 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.867417 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:55:51.868722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.868700 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:55:51.896420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.895974 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:55:51.941683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:51.939040 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:55:52.045022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.044992 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:52.046871 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.046844 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 20:55:52.324836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.324721 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" event={"ID":"5d763b0d-f27d-4fb8-becd-54cff6e1fb91","Type":"ContainerStarted","Data":"cc9851d04b76b162ed3e732300b7251deaadf30102de5870d6d20866750cbfb7"} Oct 30 20:55:52.324836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.324793 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" event={"ID":"5d763b0d-f27d-4fb8-becd-54cff6e1fb91","Type":"ContainerStarted","Data":"a3f59ceef85c13ea64e9c7e94992250aa7dafd3dd90fd5fb33fb6eb06ab138be"} Oct 30 20:55:52.337621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.337589 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:52.372425 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:52.372362 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" podStartSLOduration=6.37234576 podStartE2EDuration="6.37234576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:52.369483026 +0000 UTC m=+965.728195771" watchObservedRunningTime="2026-10-30 20:55:52.37234576 +0000 UTC m=+965.731058584" Oct 30 20:55:53.343738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:53.343303 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:54.347033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:54.347004 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:54.347993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:54.347971 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:55:55.304684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.301528 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:55:55.304684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.302455 5855 cpu_manager.go:414] "containerMap: removing container" podUID="05177ebd-27f4-479a-b459-816cb238dc4b" containerName="revoke-keys" Oct 30 20:55:55.304684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.302476 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05177ebd-27f4-479a-b459-816cb238dc4b" containerName="revoke-keys" Oct 30 20:55:55.304684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.302731 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="05177ebd-27f4-479a-b459-816cb238dc4b" containerName="revoke-keys" Oct 30 20:55:55.307491 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.307470 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.320806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.320771 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:55:55.410759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.410407 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ab941206\"" Oct 30 20:55:55.410759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.410647 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ab941206-dockercfg-gcbvg\"" Oct 30 20:55:55.678737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.637492 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.734495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.734453 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcwf\" (UniqueName: \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.860679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.841221 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.860679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.841547 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcwf\" (UniqueName: \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.904696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.888462 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcwf\" (UniqueName: \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:55.918357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:55.918300 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:56.028750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:56.026334 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:56.911302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:56.911275 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:55:57.138569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.138534 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:55:57.143552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.143524 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.153482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.153453 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:55:57.223786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.188047 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a8daf029-dockercfg-bgrxs\"" Oct 30 20:55:57.225460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.225398 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a8daf029\"" Oct 30 20:55:57.251220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.249212 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.251220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.249342 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqz4g\" (UniqueName: \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.353073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.353024 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.353487 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.353467 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz4g\" (UniqueName: \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.365848 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.360118 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.424372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.424331 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqz4g\" (UniqueName: \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\") pod \"payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.628691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.599517 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:57.922105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.922063 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" event={"ID":"7b28c334-013d-4173-83be-3b3b722a8688","Type":"ContainerStarted","Data":"9e71d773e8e93db4521eedb6b73bd7ce3e7786f44dadf5c32a124a4949eeed0d"} Oct 30 20:55:57.922565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.922130 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" event={"ID":"7b28c334-013d-4173-83be-3b3b722a8688","Type":"ContainerStarted","Data":"80234e9f9a5bce830e9459985ae235212c38d398846bf95077d2cb81033e9386"} Oct 30 20:55:57.922565 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.922160 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:55:57.963118 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:57.963057 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" podStartSLOduration=2.9630369 podStartE2EDuration="2.9630369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:57.943414793 +0000 UTC m=+971.302127536" watchObservedRunningTime="2026-10-30 20:55:57.9630369 +0000 UTC m=+971.321749649" Oct 30 20:55:58.203266 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:58.203236 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:55:58.933192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:58.933158 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" event={"ID":"8cbb4fad-696d-4040-9e6d-733ce66598d5","Type":"ContainerStarted","Data":"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49"} Oct 30 20:55:58.933632 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:58.933213 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" event={"ID":"8cbb4fad-696d-4040-9e6d-733ce66598d5","Type":"ContainerStarted","Data":"d2a2448f43e98eeac925ea175b64c0c95b0f748bd50c90fafb54ea84433c6901"} Oct 30 20:55:58.933632 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:58.933276 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:55:58.953870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:55:58.953799 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" podStartSLOduration=1.953775987 podStartE2EDuration="1.953775987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:58.95190542 +0000 UTC m=+972.310618168" watchObservedRunningTime="2026-10-30 20:55:58.953775987 +0000 UTC m=+972.312488750" Oct 30 20:56:00.937445 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:56:00.937402 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:56:08.936492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:08.936457 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:56:09.952512 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:09.952480 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:56:12.104168 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:12.099230 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:56:14.642543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.642501 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr"] Oct 30 20:56:14.660513 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.660482 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr"] Oct 30 20:56:14.660905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.660826 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.694685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.691600 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:56:14.694685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.692849 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:56:14.701717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.696019 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:14.709688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.708369 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:56:14.746956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.727398 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:56:14.746956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.727641 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:14.761691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.753802 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.761691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.754018 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.761691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.754077 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pzs\" (UniqueName: \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.874716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.865889 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.874716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.866103 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.874716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.866164 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pzs\" (UniqueName: \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:14.892760 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:14.892670 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:15.249074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.248976 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:15.254437 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.254397 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:56:15.285684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.267430 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pzs\" (UniqueName: \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:15.358683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.327794 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:56:15.358683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.329635 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06fc8a20-xnxfr\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:15.358939 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.358780 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:15.406710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.389826 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.483616 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.483576 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.501184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.501091 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.516692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.513371 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.549947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.549906 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.721365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.721316 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.722675 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.722631 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:56:15.729768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.729734 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:56:15.731679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.730488 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:56:15.993594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:15.993042 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.264741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.259800 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.264741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.261996 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.338717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.338647 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:16.339248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.339220 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:56:16.339948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.339925 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:56:16.340155 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.340139 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:56:16.342564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.342535 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.359438 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.358073 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Oct 30 20:56:16.360092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.359774 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:56:16.583754 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.533670 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.585456 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.586819 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.586862 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.647070 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.647139 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.724788 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:16.982944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.733997 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.996692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:16.995235 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145045 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145116 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145153 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145190 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145230 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:56:17.145683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.145261 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:56:17.620345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.620282 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:17.705532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.635979 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:56:17.705532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.703637 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:56:17.936744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.919953 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:17.937763 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:17.937718 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.066375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.066323 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.272774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.267793 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:18.272774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.267882 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.335796 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.335749 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:18.337718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.337637 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:56:18.339675 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.339614 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:18.350194 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.350146 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:56:18.350896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.350864 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:18.354357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.354311 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:18.355000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.354966 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:56:18.355513 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.355488 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:56:18.506497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.419223 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.506497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.488856 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:56:18.506497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.437617 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.515954 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.515909 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:56:18.736627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.736576 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.744281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:18.744240 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:56:19.148114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.148059 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:56:19.302899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.302845 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:19.303234 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.303181 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.341829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.341782 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.342042 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.342019 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.342373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.342340 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.360115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.360068 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:56:19.360776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.360743 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:19.361255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.361232 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:56:19.361635 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.361612 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:19.368020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.361788 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:19.490917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.490813 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.531784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.492463 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:56:19.532222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.532185 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:56:19.629318 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.629268 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Oct 30 20:56:19.747911 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.747812 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr"] Oct 30 20:56:19.748244 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.748212 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.750684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.750631 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.792690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:19.788281 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" event={"ID":"6dff7848-04cc-4a2a-96e5-a5e64af18c30","Type":"ContainerStarted","Data":"f9ef98a38af1a2bbd5946d4caba91effe1e9cd62adeaa763bf3e45fe799da10a"} Oct 30 20:56:20.064773 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.064268 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 20:56:20.303682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.301182 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.363742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.362746 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:20.363742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.363151 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:56:20.369722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.364682 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:20.369722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.365007 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:56:20.385721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.375797 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.385721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.376223 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.470919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.470878 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:56:20.495878 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.495820 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.589748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.544723 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.658720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.590040 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.757218 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.755840 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Oct 30 20:56:20.790326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.785549 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:56:20.873798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:20.797840 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.319106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:21.305206 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:56:22.033259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.032155 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:22.046714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.040641 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:56:22.055972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.055926 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:22.090720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.062113 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:56:22.090720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.063347 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:22.090720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.085556 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 20:56:22.225180 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.225116 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.320587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.277826 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 30 20:56:22.320587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.308291 5855 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-dck4k container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:22.320587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.308371 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-dck4k" podUID="1912739c-4112-436d-9000-e5858dce9aeb" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.13:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.397987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.397940 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:56:22.807744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.614965 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:56:22.836646 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.615205 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:56:22.837162 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.615818 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:56:22.850712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.615948 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Oct 30 20:56:22.851215 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.616139 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.879981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:22.879931 5855 generic.go:356] "Generic (PLEG): container finished" podID="6dff7848-04cc-4a2a-96e5-a5e64af18c30" containerID="aef99de9bb73142ddde3ee24741217c2552bf55bf433785ebbd55c7e96d723dc" exitCode=0 Oct 30 20:56:23.008880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.008837 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:56:23.096080 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.095973 5855 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:23.096344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.096311 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.105495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.101272 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:56:23.119133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.110316 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Oct 30 20:56:23.119133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.110476 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:23.119133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.116347 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:23.146810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.146748 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.179962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.179907 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.275300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.274415 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.304720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.290601 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded" Oct 30 20:56:23.558867 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:56:23.558833 5855 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.669s" Oct 30 20:56:23.592696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.588104 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" event={"ID":"6dff7848-04cc-4a2a-96e5-a5e64af18c30","Type":"ContainerDied","Data":"aef99de9bb73142ddde3ee24741217c2552bf55bf433785ebbd55c7e96d723dc"} Oct 30 20:56:23.692685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.687949 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 20:56:23.756126 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.756093 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:56:23.841880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.841775 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.896767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.842105 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:56:23.896767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.850453 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:56:23.918455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.918407 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:23.944857 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.944803 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.002054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:23.961576 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Oct 30 20:56:24.009791 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.009742 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 30 20:56:24.164943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.164835 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.205242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.205211 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:56:24.208529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.208410 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:24.209142 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.209111 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:24.322406 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.318844 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.331770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.169640 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:24.586300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.586243 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.698016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.697957 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.847762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.826813 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:56:24.872358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.855470 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.872358 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.856789 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.898164 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.892425 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Oct 30 20:56:24.968681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.946778 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:24.968681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:24.948427 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.130741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.130416 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.211562 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.210752 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:25.211562 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.211083 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 20:56:25.335713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.330179 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.335713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.332323 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.672716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.658821 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:56:25.673389 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.673366 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:56:25.707111 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.679807 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.707111 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.683223 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:56:25.746536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.746502 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65"] Oct 30 20:56:25.766865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.760371 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:25.860204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.858788 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.889713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.886425 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:25.925899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.925799 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:56:25.951464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:25.948184 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.084100 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.084065 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:26.084813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.084782 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.085297 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.085267 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.202575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.200962 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6md\" (UniqueName: \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.273681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.264762 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 20:56:26.359429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.359384 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.425919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.425879 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65"] Oct 30 20:56:26.426331 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.426309 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:56:26.426533 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.426517 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 20:56:26.458113 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.458024 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\") pod \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " Oct 30 20:56:26.474798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.458481 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.475302 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.475269 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:56:26.477263 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.477216 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.530754 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.510031 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\") pod \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " Oct 30 20:56:26.530754 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.510089 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\" (UniqueName: \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\") pod \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\" (UID: \"6dff7848-04cc-4a2a-96e5-a5e64af18c30\") " Oct 30 20:56:26.553726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.548694 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6md\" (UniqueName: \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.553726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.548909 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.553726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.549158 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.578272 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.563099 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.581750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.579425 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca" pod "6dff7848-04cc-4a2a-96e5-a5e64af18c30" (UID: "6dff7848-04cc-4a2a-96e5-a5e64af18c30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:26.635769 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.635708 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.749738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.740782 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 20:56:26.749738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.743453 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:56:26.810251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.810215 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs" pod "6dff7848-04cc-4a2a-96e5-a5e64af18c30" (UID: "6dff7848-04cc-4a2a-96e5-a5e64af18c30"). InnerVolumeSpecName "kube-api-access-w4pzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:26.831104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.811044 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.833627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.827540 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:26.834720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.829339 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6md\" (UniqueName: \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\") pod \"maas-api-revoke-keys-e2e-ait-ab941206-r6l65\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.860730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.860540 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:26.893674 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.893618 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:56:26.904814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.902512 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca" pod "6dff7848-04cc-4a2a-96e5-a5e64af18c30" (UID: "6dff7848-04cc-4a2a-96e5-a5e64af18c30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:26.950777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.943787 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4pzs\" (UniqueName: \"kubernetes.io/projected/6dff7848-04cc-4a2a-96e5-a5e64af18c30-kube-api-access-w4pzs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:26.950777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.943821 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dff7848-04cc-4a2a-96e5-a5e64af18c30-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:26.950777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.943965 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:26.970064 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970029 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:56:26.970326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970312 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:56:26.970455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970444 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:56:26.970579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970568 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl"] Oct 30 20:56:26.970712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970700 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:56:26.970838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970827 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:56:26.970962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.970949 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 20:56:26.981558 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.981528 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dae7b1d-rhfgl"] Oct 30 20:56:26.983818 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.983797 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" Oct 30 20:56:26.986771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.986745 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr" event={"ID":"6dff7848-04cc-4a2a-96e5-a5e64af18c30","Type":"ContainerDied","Data":"f9ef98a38af1a2bbd5946d4caba91effe1e9cd62adeaa763bf3e45fe799da10a"} Oct 30 20:56:26.987749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:26.986938 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ef98a38af1a2bbd5946d4caba91effe1e9cd62adeaa763bf3e45fe799da10a" Oct 30 20:56:27.404679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:27.387194 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65"] Oct 30 20:56:28.053248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:28.053201 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" event={"ID":"2e02a8ed-ea63-4233-be77-c12f8c220014","Type":"ContainerStarted","Data":"5bccff62b1b943f526028f19eeb025f027e9d4b245e6d44681e6093c8eedef0d"} Oct 30 20:56:28.053706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:28.053266 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" event={"ID":"2e02a8ed-ea63-4233-be77-c12f8c220014","Type":"ContainerStarted","Data":"10ddc51c8787ac78f3c3c9c67e17f8aa055e4abeb1b239f28fcd1dcaa2dcccb3"} Oct 30 20:56:28.079625 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:28.079565 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" podStartSLOduration=4.079546114 podStartE2EDuration="4.079546114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:28.071996218 +0000 UTC m=+1001.430708983" watchObservedRunningTime="2026-10-30 20:56:28.079546114 +0000 UTC m=+1001.438258861" Oct 30 20:56:28.868681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:28.866834 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1b51ad-6177-405d-ac18-b94e2da9ff90" path="/var/lib/kubelet/pods/3c1b51ad-6177-405d-ac18-b94e2da9ff90/volumes" Oct 30 20:56:29.085222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.084524 5855 generic.go:356] "Generic (PLEG): container finished" podID="2e02a8ed-ea63-4233-be77-c12f8c220014" containerID="5bccff62b1b943f526028f19eeb025f027e9d4b245e6d44681e6093c8eedef0d" exitCode=0 Oct 30 20:56:29.085222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.084729 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" event={"ID":"2e02a8ed-ea63-4233-be77-c12f8c220014","Type":"ContainerDied","Data":"5bccff62b1b943f526028f19eeb025f027e9d4b245e6d44681e6093c8eedef0d"} Oct 30 20:56:29.974965 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.974926 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 20:56:29.975626 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.975612 5855 cpu_manager.go:414] "containerMap: removing container" podUID="6dff7848-04cc-4a2a-96e5-a5e64af18c30" containerName="revoke-keys" Oct 30 20:56:29.975685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.975627 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dff7848-04cc-4a2a-96e5-a5e64af18c30" containerName="revoke-keys" Oct 30 20:56:29.975776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.975765 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dff7848-04cc-4a2a-96e5-a5e64af18c30" containerName="revoke-keys" Oct 30 20:56:29.979870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.979844 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.997882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.997851 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998038 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.997901 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.998126 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.998165 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4hcb\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998284 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.998267 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998350 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.998334 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:29.998413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:29.998400 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.012440 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.012399 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 20:56:30.100254 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100224 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100614 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100266 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100614 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100442 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100614 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100493 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100667 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100699 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4hcb\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.100853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.100762 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.102586 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.102552 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.105345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.105317 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.105591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.105570 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.105701 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.105681 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.107441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.107395 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.114786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.114725 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.115803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.115770 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4hcb\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b6fwv\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.302398 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.293556 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:30.438735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.438301 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:30.512527 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.512476 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\") pod \"2e02a8ed-ea63-4233-be77-c12f8c220014\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " Oct 30 20:56:30.529843 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.529809 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\") pod \"2e02a8ed-ea63-4233-be77-c12f8c220014\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " Oct 30 20:56:30.530174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.530150 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\" (UniqueName: \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\") pod \"2e02a8ed-ea63-4233-be77-c12f8c220014\" (UID: \"2e02a8ed-ea63-4233-be77-c12f8c220014\") " Oct 30 20:56:30.555696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.553749 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca" pod "2e02a8ed-ea63-4233-be77-c12f8c220014" (UID: "2e02a8ed-ea63-4233-be77-c12f8c220014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:30.594734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.586876 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca" pod "2e02a8ed-ea63-4233-be77-c12f8c220014" (UID: "2e02a8ed-ea63-4233-be77-c12f8c220014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:30.594734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.591219 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md" pod "2e02a8ed-ea63-4233-be77-c12f8c220014" (UID: "2e02a8ed-ea63-4233-be77-c12f8c220014"). InnerVolumeSpecName "kube-api-access-dk6md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:30.696959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.694435 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:56:30.696959 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.694523 5855 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 20:56:30.703801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.698220 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:30.703801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.698362 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e02a8ed-ea63-4233-be77-c12f8c220014-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:30.703801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.698384 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6md\" (UniqueName: \"kubernetes.io/projected/2e02a8ed-ea63-4233-be77-c12f8c220014-kube-api-access-dk6md\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:30.711231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:30.711189 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 20:56:30.929864 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:56:30.929766 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:56:31.116542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.116500 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" event={"ID":"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6","Type":"ContainerStarted","Data":"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3"} Oct 30 20:56:31.117119 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.116564 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" event={"ID":"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6","Type":"ContainerStarted","Data":"5b6fa019d79749a46e544337726499934085b757266350ae8451e8d7022796ba"} Oct 30 20:56:31.117119 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.116881 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:31.119184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.118971 5855 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b6fwv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:56:31.119443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.119415 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:56:31.119579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.119545 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" event={"ID":"2e02a8ed-ea63-4233-be77-c12f8c220014","Type":"ContainerDied","Data":"10ddc51c8787ac78f3c3c9c67e17f8aa055e4abeb1b239f28fcd1dcaa2dcccb3"} Oct 30 20:56:31.119579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.119570 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65" Oct 30 20:56:31.119685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.119602 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ddc51c8787ac78f3c3c9c67e17f8aa055e4abeb1b239f28fcd1dcaa2dcccb3" Oct 30 20:56:31.169289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:31.169229 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" podStartSLOduration=2.169214168 podStartE2EDuration="2.169214168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:31.167904484 +0000 UTC m=+1004.526617237" watchObservedRunningTime="2026-10-30 20:56:31.169214168 +0000 UTC m=+1004.527926913" Oct 30 20:56:32.130688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:32.130556 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 20:56:35.142494 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:35.142454 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:56:36.049532 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:36.049492 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4"] Oct 30 20:56:36.068104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:36.068051 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c9e285-t8fq4"] Oct 30 20:56:36.837991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:36.837950 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1480c7a-756b-4550-b917-e5ad20542517" path="/var/lib/kubelet/pods/f1480c7a-756b-4550-b917-e5ad20542517/volumes" Oct 30 20:56:40.143806 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:40.143757 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" podUID="7b28c334-013d-4173-83be-3b3b722a8688" containerName="payload-pre-processing" containerID="cri-o://9e71d773e8e93db4521eedb6b73bd7ce3e7786f44dadf5c32a124a4949eeed0d" gracePeriod=25 Oct 30 20:56:40.182014 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:40.181962 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65"] Oct 30 20:56:40.187941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:40.187903 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ab941206-r6l65"] Oct 30 20:56:41.139122 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.139084 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e02a8ed-ea63-4233-be77-c12f8c220014" path="/var/lib/kubelet/pods/2e02a8ed-ea63-4233-be77-c12f8c220014/volumes" Oct 30 20:56:41.253418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.253387 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9"] Oct 30 20:56:41.255180 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.255153 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2e02a8ed-ea63-4233-be77-c12f8c220014" containerName="revoke-keys" Oct 30 20:56:41.255338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.255326 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e02a8ed-ea63-4233-be77-c12f8c220014" containerName="revoke-keys" Oct 30 20:56:41.255736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.255710 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e02a8ed-ea63-4233-be77-c12f8c220014" containerName="revoke-keys" Oct 30 20:56:41.264932 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.264904 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.270600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.270566 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:56:41.271871 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.271769 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:56:41.272114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.272100 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:56:41.272259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.272249 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:56:41.307711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.307574 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9"] Oct 30 20:56:41.332083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.332027 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:56:41.332684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.332602 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" podUID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" containerName="istio-proxy" containerID="cri-o://cc9851d04b76b162ed3e732300b7251deaadf30102de5870d6d20866750cbfb7" gracePeriod=30 Oct 30 20:56:41.444681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.444609 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.445879 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.445731 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.445879 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.445792 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwl2\" (UniqueName: \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.510727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.503182 5855 generic.go:356] "Generic (PLEG): container finished" podID="7b28c334-013d-4173-83be-3b3b722a8688" containerID="9e71d773e8e93db4521eedb6b73bd7ce3e7786f44dadf5c32a124a4949eeed0d" exitCode=0 Oct 30 20:56:41.510727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.503429 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" event={"ID":"7b28c334-013d-4173-83be-3b3b722a8688","Type":"ContainerDied","Data":"9e71d773e8e93db4521eedb6b73bd7ce3e7786f44dadf5c32a124a4949eeed0d"} Oct 30 20:56:41.510727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.503499 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" event={"ID":"7b28c334-013d-4173-83be-3b3b722a8688","Type":"ContainerDied","Data":"80234e9f9a5bce830e9459985ae235212c38d398846bf95077d2cb81033e9386"} Oct 30 20:56:41.510727 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.503524 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80234e9f9a5bce830e9459985ae235212c38d398846bf95077d2cb81033e9386" Oct 30 20:56:41.546813 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.531641 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:56:41.581177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.574837 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.581177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.574912 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwl2\" (UniqueName: \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.581177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.575155 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.581177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.576202 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.699049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.698963 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\" (UniqueName: \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\") pod \"7b28c334-013d-4173-83be-3b3b722a8688\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " Oct 30 20:56:41.699342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.699322 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\") pod \"7b28c334-013d-4173-83be-3b3b722a8688\" (UID: \"7b28c334-013d-4173-83be-3b3b722a8688\") " Oct 30 20:56:41.708185 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.707925 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwl2\" (UniqueName: \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.709048 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.709020 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume" pod "7b28c334-013d-4173-83be-3b3b722a8688" (UID: "7b28c334-013d-4173-83be-3b3b722a8688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:41.799567 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.799494 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf" pod "7b28c334-013d-4173-83be-3b3b722a8688" (UID: "7b28c334-013d-4173-83be-3b3b722a8688"). InnerVolumeSpecName "kube-api-access-4vcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:41.803785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.801243 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:41.804509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.803992 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vcwf\" (UniqueName: \"kubernetes.io/projected/7b28c334-013d-4173-83be-3b3b722a8688-kube-api-access-4vcwf\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:41.804509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.804037 5855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b28c334-013d-4173-83be-3b3b722a8688-plugins-config-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:41.804509 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:41.804239 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:42.099869 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:56:42.099834 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941061dc_95bb_4758_8ab4_fedc9e4c0685.slice/crio-f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd WatchSource:0}: Error finding container f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd: Status 404 returned error can't find the container with id f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd Oct 30 20:56:42.101906 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.101868 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9"] Oct 30 20:56:42.380056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.379969 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2"] Oct 30 20:56:42.381558 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.381527 5855 cpu_manager.go:414] "containerMap: removing container" podUID="7b28c334-013d-4173-83be-3b3b722a8688" containerName="payload-pre-processing" Oct 30 20:56:42.381729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.381560 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b28c334-013d-4173-83be-3b3b722a8688" containerName="payload-pre-processing" Oct 30 20:56:42.381778 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.381754 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b28c334-013d-4173-83be-3b3b722a8688" containerName="payload-pre-processing" Oct 30 20:56:42.389713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.387933 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.396115 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.395870 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2"] Oct 30 20:56:42.522103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.522012 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" event={"ID":"941061dc-95bb-4758-8ab4-fedc9e4c0685","Type":"ContainerStarted","Data":"ff5795d27b99f0b53fe3e7698ba78d1cde33e4251b84566def5f415b8f13872a"} Oct 30 20:56:42.522465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.522132 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4" Oct 30 20:56:42.522465 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.522135 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" event={"ID":"941061dc-95bb-4758-8ab4-fedc9e4c0685","Type":"ContainerStarted","Data":"f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd"} Oct 30 20:56:42.529103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.529021 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.529451 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.529347 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzbd\" (UniqueName: \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.530094 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.530056 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.558186 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.558078 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" podStartSLOduration=2.558044576 podStartE2EDuration="2.558044576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:42.549622397 +0000 UTC m=+1015.908335178" watchObservedRunningTime="2026-10-30 20:56:42.558044576 +0000 UTC m=+1015.916757323" Oct 30 20:56:42.582260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.582186 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:56:42.592445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.592349 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab941206-5569d689bc-ch9l4"] Oct 30 20:56:42.631464 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.631376 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzbd\" (UniqueName: \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.631647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.631578 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.631788 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.631714 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.632329 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.632299 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.634286 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.634264 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.645504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.645469 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzbd\" (UniqueName: \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\") pod \"maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.714480 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.714449 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:42.802593 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:42.802561 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b28c334-013d-4173-83be-3b3b722a8688" path="/var/lib/kubelet/pods/7b28c334-013d-4173-83be-3b3b722a8688/volumes" Oct 30 20:56:43.010228 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:56:43.010192 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22747d28_8b30_48fb_8b55_b3b254a0024f.slice/crio-5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1 WatchSource:0}: Error finding container 5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1: Status 404 returned error can't find the container with id 5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1 Oct 30 20:56:43.011942 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.011911 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2"] Oct 30 20:56:43.553594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.553553 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" event={"ID":"22747d28-8b30-48fb-8b55-b3b254a0024f","Type":"ContainerStarted","Data":"f495fe9df4312f8454c9a0a217a399c9ff9a78b2c130ac15a200e437cc81c288"} Oct 30 20:56:43.554258 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.554232 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" event={"ID":"22747d28-8b30-48fb-8b55-b3b254a0024f","Type":"ContainerStarted","Data":"5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1"} Oct 30 20:56:43.579798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.575134 5855 generic.go:356] "Generic (PLEG): container finished" podID="941061dc-95bb-4758-8ab4-fedc9e4c0685" containerID="ff5795d27b99f0b53fe3e7698ba78d1cde33e4251b84566def5f415b8f13872a" exitCode=0 Oct 30 20:56:43.580073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.578772 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" event={"ID":"941061dc-95bb-4758-8ab4-fedc9e4c0685","Type":"ContainerDied","Data":"ff5795d27b99f0b53fe3e7698ba78d1cde33e4251b84566def5f415b8f13872a"} Oct 30 20:56:43.593693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:43.592978 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" podStartSLOduration=1.592946978 podStartE2EDuration="1.592946978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:43.590065414 +0000 UTC m=+1016.948778163" watchObservedRunningTime="2026-10-30 20:56:43.592946978 +0000 UTC m=+1016.951659726" Oct 30 20:56:44.603355 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.603316 5855 generic.go:356] "Generic (PLEG): container finished" podID="22747d28-8b30-48fb-8b55-b3b254a0024f" containerID="f495fe9df4312f8454c9a0a217a399c9ff9a78b2c130ac15a200e437cc81c288" exitCode=0 Oct 30 20:56:44.603957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.603406 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" event={"ID":"22747d28-8b30-48fb-8b55-b3b254a0024f","Type":"ContainerDied","Data":"f495fe9df4312f8454c9a0a217a399c9ff9a78b2c130ac15a200e437cc81c288"} Oct 30 20:56:44.792136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.792102 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:44.951740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.950886 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\" (UniqueName: \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\") pod \"941061dc-95bb-4758-8ab4-fedc9e4c0685\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " Oct 30 20:56:44.951740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.950980 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\") pod \"941061dc-95bb-4758-8ab4-fedc9e4c0685\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " Oct 30 20:56:44.951740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:44.951188 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\") pod \"941061dc-95bb-4758-8ab4-fedc9e4c0685\" (UID: \"941061dc-95bb-4758-8ab4-fedc9e4c0685\") " Oct 30 20:56:45.032680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.029250 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca" pod "941061dc-95bb-4758-8ab4-fedc9e4c0685" (UID: "941061dc-95bb-4758-8ab4-fedc9e4c0685"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:45.040905 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.038539 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca" pod "941061dc-95bb-4758-8ab4-fedc9e4c0685" (UID: "941061dc-95bb-4758-8ab4-fedc9e4c0685"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:45.067044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.067010 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2" pod "941061dc-95bb-4758-8ab4-fedc9e4c0685" (UID: "941061dc-95bb-4758-8ab4-fedc9e4c0685"). InnerVolumeSpecName "kube-api-access-jlwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:45.143682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.093807 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwl2\" (UniqueName: \"kubernetes.io/projected/941061dc-95bb-4758-8ab4-fedc9e4c0685-kube-api-access-jlwl2\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:45.143682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.093849 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:45.143682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.093866 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/941061dc-95bb-4758-8ab4-fedc9e4c0685-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:45.661204 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.655813 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" Oct 30 20:56:45.662190 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.661802 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9" event={"ID":"941061dc-95bb-4758-8ab4-fedc9e4c0685","Type":"ContainerDied","Data":"f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd"} Oct 30 20:56:45.662190 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.661884 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f51076ef883698ef769c8fddf7ec0fd867ea70fb59cb398c58f3c34c1fe6d8cd" Oct 30 20:56:45.917734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:45.914971 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:46.025234 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.025183 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\") pod \"22747d28-8b30-48fb-8b55-b3b254a0024f\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " Oct 30 20:56:46.025946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.025889 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\") pod \"22747d28-8b30-48fb-8b55-b3b254a0024f\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " Oct 30 20:56:46.026285 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.026160 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\" (UniqueName: \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\") pod \"22747d28-8b30-48fb-8b55-b3b254a0024f\" (UID: \"22747d28-8b30-48fb-8b55-b3b254a0024f\") " Oct 30 20:56:46.028100 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.028074 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca" pod "22747d28-8b30-48fb-8b55-b3b254a0024f" (UID: "22747d28-8b30-48fb-8b55-b3b254a0024f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:46.034294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.034272 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca" pod "22747d28-8b30-48fb-8b55-b3b254a0024f" (UID: "22747d28-8b30-48fb-8b55-b3b254a0024f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:56:46.041105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.041055 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd" pod "22747d28-8b30-48fb-8b55-b3b254a0024f" (UID: "22747d28-8b30-48fb-8b55-b3b254a0024f"). InnerVolumeSpecName "kube-api-access-rbzbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:46.129054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.129017 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:46.129298 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.129283 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzbd\" (UniqueName: \"kubernetes.io/projected/22747d28-8b30-48fb-8b55-b3b254a0024f-kube-api-access-rbzbd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:46.129379 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.129370 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22747d28-8b30-48fb-8b55-b3b254a0024f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:46.687744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.681523 5855 generic.go:356] "Generic (PLEG): container finished" podID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" containerID="cc9851d04b76b162ed3e732300b7251deaadf30102de5870d6d20866750cbfb7" exitCode=0 Oct 30 20:56:46.687744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.681907 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" event={"ID":"5d763b0d-f27d-4fb8-becd-54cff6e1fb91","Type":"ContainerDied","Data":"cc9851d04b76b162ed3e732300b7251deaadf30102de5870d6d20866750cbfb7"} Oct 30 20:56:46.687744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.685484 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" Oct 30 20:56:46.694873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.694761 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2" event={"ID":"22747d28-8b30-48fb-8b55-b3b254a0024f","Type":"ContainerDied","Data":"5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1"} Oct 30 20:56:46.694873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:46.694830 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1d6ab1015a6e77ba879c1f9fca0cbf324baca0e523b66190f4bafa9afe10a1" Oct 30 20:56:47.539758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.532675 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:56:47.657979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.657580 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.658309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.658284 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.658468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.658452 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.658831 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.658810 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.659081 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.659063 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.662749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.662151 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:47.664472 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.663955 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:47.664620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.664534 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.666594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.666530 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:47.681165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.680866 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:47.681165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.681025 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.681165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.681164 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.681468 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.681200 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\") pod \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\" (UID: \"5d763b0d-f27d-4fb8-becd-54cff6e1fb91\") " Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.689055 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.710072 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.710112 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.710130 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.710144 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.715694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.710158 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.882722 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.870846 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "kube-api-access-v7d7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:47.956599 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.956560 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7d7x\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-kube-api-access-v7d7x\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:47.968405 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.968328 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:56:48.003682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.983460 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:56:48.003682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:47.995023 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token" pod "5d763b0d-f27d-4fb8-becd-54cff6e1fb91" (UID: "5d763b0d-f27d-4fb8-becd-54cff6e1fb91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:48.028680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.021798 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" event={"ID":"5d763b0d-f27d-4fb8-becd-54cff6e1fb91","Type":"ContainerDied","Data":"a3f59ceef85c13ea64e9c7e94992250aa7dafd3dd90fd5fb33fb6eb06ab138be"} Oct 30 20:56:48.028680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.021881 5855 scope.go:122] "RemoveContainer" containerID="cc9851d04b76b162ed3e732300b7251deaadf30102de5870d6d20866750cbfb7" Oct 30 20:56:48.028680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.022056 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn" Oct 30 20:56:48.081362 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.070946 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:48.081362 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.070981 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:48.081362 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.070997 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d763b0d-f27d-4fb8-becd-54cff6e1fb91-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:48.256683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.256617 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:56:48.331550 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.316991 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ab941206-openshift-default-7b5bc78467-jmkkn"] Oct 30 20:56:48.935900 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.907327 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" path="/var/lib/kubelet/pods/5d763b0d-f27d-4fb8-becd-54cff6e1fb91/volumes" Oct 30 20:56:48.997685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.997031 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:48.997685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:48.997110 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.312785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:49.310344 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf"] Oct 30 20:56:49.312785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:49.311143 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c3e573-x4wnf"] Oct 30 20:56:50.043470 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:50.043437 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x"] Oct 30 20:56:50.050972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:50.050935 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0edf61d9-28z7x"] Oct 30 20:56:50.783178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:50.783141 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03bea02-9a84-47a8-894a-62edc8a9dcf8" path="/var/lib/kubelet/pods/a03bea02-9a84-47a8-894a-62edc8a9dcf8/volumes" Oct 30 20:56:50.783827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:50.783809 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4b1c99-451c-4577-8416-42082f152425" path="/var/lib/kubelet/pods/ae4b1c99-451c-4577-8416-42082f152425/volumes" Oct 30 20:56:52.902485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:52.902452 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:56:53.396213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:53.396166 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:53.397120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:53.396685 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:53.576044 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:53.568897 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:54.225684 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:54.220716 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9"] Oct 30 20:56:54.281682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:54.280828 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2e92a84c-wllf9"] Oct 30 20:56:54.539558 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:54.539463 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:56:54.562836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:54.562797 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:56:54.962492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:54.962458 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941061dc-95bb-4758-8ab4-fedc9e4c0685" path="/var/lib/kubelet/pods/941061dc-95bb-4758-8ab4-fedc9e4c0685/volumes" Oct 30 20:56:55.459242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.459206 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf"] Oct 30 20:56:55.459993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.459970 5855 cpu_manager.go:414] "containerMap: removing container" podUID="941061dc-95bb-4758-8ab4-fedc9e4c0685" containerName="revoke-keys" Oct 30 20:56:55.459993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.459995 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941061dc-95bb-4758-8ab4-fedc9e4c0685" containerName="revoke-keys" Oct 30 20:56:55.460128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460010 5855 cpu_manager.go:414] "containerMap: removing container" podUID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" containerName="istio-proxy" Oct 30 20:56:55.460128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460015 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" containerName="istio-proxy" Oct 30 20:56:55.460128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460043 5855 cpu_manager.go:414] "containerMap: removing container" podUID="22747d28-8b30-48fb-8b55-b3b254a0024f" containerName="revoke-keys" Oct 30 20:56:55.460128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460047 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22747d28-8b30-48fb-8b55-b3b254a0024f" containerName="revoke-keys" Oct 30 20:56:55.460303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460200 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="22747d28-8b30-48fb-8b55-b3b254a0024f" containerName="revoke-keys" Oct 30 20:56:55.460303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460216 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="941061dc-95bb-4758-8ab4-fedc9e4c0685" containerName="revoke-keys" Oct 30 20:56:55.460303 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.460222 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d763b0d-f27d-4fb8-becd-54cff6e1fb91" containerName="istio-proxy" Oct 30 20:56:55.466255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.466227 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.484740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.477806 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf"] Oct 30 20:56:55.498817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.491496 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:56:55.510452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.499864 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:56:55.510928 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.510890 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:56:55.511287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.511260 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:56:55.617123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.617074 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7t7\" (UniqueName: \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.617343 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.617270 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.617577 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.617559 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.719123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.719030 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.719330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.719311 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.719436 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.719420 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7t7\" (UniqueName: \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.720332 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.720296 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.721840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.721820 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.732518 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.732490 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7t7\" (UniqueName: \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\") pod \"maas-api-revoke-sub-972fcca5-db8cf\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:55.830572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:55.830539 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:56:56.204153 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:56.204114 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf"] Oct 30 20:56:56.699820 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:56.699779 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" event={"ID":"e15da1b9-a128-4942-a8c2-d33db7c92c1a","Type":"ContainerStarted","Data":"1dc2b173d0743deacc96a05a0822a4f7faf0cd441baf18f79312f5caf49b2752"} Oct 30 20:56:56.700325 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:56.699846 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" event={"ID":"e15da1b9-a128-4942-a8c2-d33db7c92c1a","Type":"ContainerStarted","Data":"aa5f0ccc49da6d8dd150520f0fa614777dae2908794cb64ac672244551afd074"} Oct 30 20:56:56.727236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:56.727173 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" podStartSLOduration=1.7271502079999999 podStartE2EDuration="1.727150208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:56.720870187 +0000 UTC m=+1030.079582945" watchObservedRunningTime="2026-10-30 20:56:56.727150208 +0000 UTC m=+1030.085862957" Oct 30 20:56:57.820295 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:57.820244 5855 generic.go:356] "Generic (PLEG): container finished" podID="e15da1b9-a128-4942-a8c2-d33db7c92c1a" containerID="1dc2b173d0743deacc96a05a0822a4f7faf0cd441baf18f79312f5caf49b2752" exitCode=0 Oct 30 20:56:57.820985 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:57.820958 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" event={"ID":"e15da1b9-a128-4942-a8c2-d33db7c92c1a","Type":"ContainerDied","Data":"1dc2b173d0743deacc96a05a0822a4f7faf0cd441baf18f79312f5caf49b2752"} Oct 30 20:56:57.923682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:57.910788 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" podUID="8cbb4fad-696d-4040-9e6d-733ce66598d5" containerName="payload-processing" containerID="cri-o://adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49" gracePeriod=25 Oct 30 20:56:58.667982 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:58.654461 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2"] Oct 30 20:56:58.697647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:58.697586 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a8daf029-2t9l2"] Oct 30 20:56:58.757682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:58.752451 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:56:59.045743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:58.926686 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\") pod \"8cbb4fad-696d-4040-9e6d-733ce66598d5\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " Oct 30 20:56:59.180685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:58.939254 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume" pod "8cbb4fad-696d-4040-9e6d-733ce66598d5" (UID: "8cbb4fad-696d-4040-9e6d-733ce66598d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:59.180685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.158103 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\" (UniqueName: \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\") pod \"8cbb4fad-696d-4040-9e6d-733ce66598d5\" (UID: \"8cbb4fad-696d-4040-9e6d-733ce66598d5\") " Oct 30 20:56:59.180685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.161327 5855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cbb4fad-696d-4040-9e6d-733ce66598d5-plugins-config-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:59.285686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.282693 5855 generic.go:356] "Generic (PLEG): container finished" podID="8cbb4fad-696d-4040-9e6d-733ce66598d5" containerID="adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49" exitCode=0 Oct 30 20:56:59.285686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.283076 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" Oct 30 20:56:59.323224 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.310028 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22747d28-8b30-48fb-8b55-b3b254a0024f" path="/var/lib/kubelet/pods/22747d28-8b30-48fb-8b55-b3b254a0024f/volumes" Oct 30 20:56:59.336105 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.336063 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" event={"ID":"8cbb4fad-696d-4040-9e6d-733ce66598d5","Type":"ContainerDied","Data":"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49"} Oct 30 20:56:59.336357 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.336338 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b" event={"ID":"8cbb4fad-696d-4040-9e6d-733ce66598d5","Type":"ContainerDied","Data":"d2a2448f43e98eeac925ea175b64c0c95b0f748bd50c90fafb54ea84433c6901"} Oct 30 20:56:59.336543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.336479 5855 scope.go:122] "RemoveContainer" containerID="adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49" Oct 30 20:56:59.369385 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.369345 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g" pod "8cbb4fad-696d-4040-9e6d-733ce66598d5" (UID: "8cbb4fad-696d-4040-9e6d-733ce66598d5"). InnerVolumeSpecName "kube-api-access-kqz4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:59.388853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.388547 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:59.483152 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.456177 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqz4g\" (UniqueName: \"kubernetes.io/projected/8cbb4fad-696d-4040-9e6d-733ce66598d5-kube-api-access-kqz4g\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:59.565429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.564785 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Oct 30 20:56:59.565830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.565795 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:59.664321 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.664293 5855 scope.go:122] "RemoveContainer" containerID="adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49" Oct 30 20:56:59.794027 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:56:59.793965 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49\": container with ID starting with adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49 not found: ID does not exist" containerID="adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49" Oct 30 20:56:59.794378 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.794335 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49"} err="failed to get container status \"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49\": rpc error: code = NotFound desc = could not find container \"adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49\": container with ID starting with adced60b5b6240640a1c17486426888b16d562db8434318cea3f87666c2d7b49 not found: ID does not exist" Oct 30 20:56:59.810699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.808783 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:59.811936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.811128 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:59.848585 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.848546 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd"] Oct 30 20:56:59.850051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.850016 5855 cpu_manager.go:414] "containerMap: removing container" podUID="8cbb4fad-696d-4040-9e6d-733ce66598d5" containerName="payload-processing" Oct 30 20:56:59.899755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.899716 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbb4fad-696d-4040-9e6d-733ce66598d5" containerName="payload-processing" Oct 30 20:56:59.900301 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.900276 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbb4fad-696d-4040-9e6d-733ce66598d5" containerName="payload-processing" Oct 30 20:56:59.947910 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:56:59.947813 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:00.097972 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.097938 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd"] Oct 30 20:57:00.109615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.109575 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:00.123774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.110048 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 20:57:00.173925 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.173887 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq"] Oct 30 20:57:00.229541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.229461 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.294306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.281617 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drm9p\" (UniqueName: \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.294306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.281867 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.294306 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.281942 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.332099 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.331998 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f218d58-jbwmq"] Oct 30 20:57:00.340282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.340255 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:57:00.349836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.349737 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a8daf029-86ccc88664-gvq5b"] Oct 30 20:57:00.383373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.383335 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.383721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.383645 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.384074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.384053 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drm9p\" (UniqueName: \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.384447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.384422 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.387380 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.387356 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.397914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.397873 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drm9p\" (UniqueName: \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\") pod \"maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.456534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.456503 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:57:00.485327 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.485236 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\") pod \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " Oct 30 20:57:00.485515 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.485454 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\") pod \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " Oct 30 20:57:00.485584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.485516 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\" (UniqueName: \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\") pod \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\" (UID: \"e15da1b9-a128-4942-a8c2-d33db7c92c1a\") " Oct 30 20:57:00.485944 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.485914 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca" pod "e15da1b9-a128-4942-a8c2-d33db7c92c1a" (UID: "e15da1b9-a128-4942-a8c2-d33db7c92c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:00.486332 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.486250 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.491738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.488092 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca" pod "e15da1b9-a128-4942-a8c2-d33db7c92c1a" (UID: "e15da1b9-a128-4942-a8c2-d33db7c92c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:00.491738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.488415 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7" pod "e15da1b9-a128-4942-a8c2-d33db7c92c1a" (UID: "e15da1b9-a128-4942-a8c2-d33db7c92c1a"). InnerVolumeSpecName "kube-api-access-dt7t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:00.587691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.587614 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7t7\" (UniqueName: \"kubernetes.io/projected/e15da1b9-a128-4942-a8c2-d33db7c92c1a-kube-api-access-dt7t7\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.587871 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.587733 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e15da1b9-a128-4942-a8c2-d33db7c92c1a-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.634990 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.634948 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:00.865683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.864307 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbb4fad-696d-4040-9e6d-733ce66598d5" path="/var/lib/kubelet/pods/8cbb4fad-696d-4040-9e6d-733ce66598d5/volumes" Oct 30 20:57:00.910183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.901450 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab33f8a3-7a8c-4565-861f-be4add5ba8d9" path="/var/lib/kubelet/pods/ab33f8a3-7a8c-4565-861f-be4add5ba8d9/volumes" Oct 30 20:57:00.951145 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:00.951118 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd"] Oct 30 20:57:00.975113 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:57:00.975079 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:57:01.358539 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.358475 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" event={"ID":"cf6e9741-0ee4-4956-9513-0b9552e6a867","Type":"ContainerStarted","Data":"e0dd2760d544ca918d286020b1f950e185d83bad0df5ed30d854753af969519d"} Oct 30 20:57:01.358539 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.358543 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" event={"ID":"cf6e9741-0ee4-4956-9513-0b9552e6a867","Type":"ContainerStarted","Data":"ddc91e1191461fd297b13b4ec8ece4107c5c4e4d317b87fa9d9def26082b867e"} Oct 30 20:57:01.361526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.361500 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" event={"ID":"e15da1b9-a128-4942-a8c2-d33db7c92c1a","Type":"ContainerDied","Data":"aa5f0ccc49da6d8dd150520f0fa614777dae2908794cb64ac672244551afd074"} Oct 30 20:57:01.361694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.361556 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5f0ccc49da6d8dd150520f0fa614777dae2908794cb64ac672244551afd074" Oct 30 20:57:01.361694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.361609 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf" Oct 30 20:57:01.381288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:01.381233 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" podStartSLOduration=2.381216431 podStartE2EDuration="2.381216431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:01.379349301 +0000 UTC m=+1034.738062047" watchObservedRunningTime="2026-10-30 20:57:01.381216431 +0000 UTC m=+1034.739929223" Oct 30 20:57:02.452183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:02.452140 5855 generic.go:356] "Generic (PLEG): container finished" podID="cf6e9741-0ee4-4956-9513-0b9552e6a867" containerID="e0dd2760d544ca918d286020b1f950e185d83bad0df5ed30d854753af969519d" exitCode=0 Oct 30 20:57:02.453123 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:02.453085 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" event={"ID":"cf6e9741-0ee4-4956-9513-0b9552e6a867","Type":"ContainerDied","Data":"e0dd2760d544ca918d286020b1f950e185d83bad0df5ed30d854753af969519d"} Oct 30 20:57:03.132718 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.132678 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt"] Oct 30 20:57:03.144519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.144476 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq"] Oct 30 20:57:03.153762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.153710 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d24306-4hxqq"] Oct 30 20:57:03.166165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.166109 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b230c2d-sfwjt"] Oct 30 20:57:03.644706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.644675 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:03.757759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.757369 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\" (UniqueName: \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\") pod \"cf6e9741-0ee4-4956-9513-0b9552e6a867\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " Oct 30 20:57:03.757759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.757436 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\") pod \"cf6e9741-0ee4-4956-9513-0b9552e6a867\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " Oct 30 20:57:03.757759 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.757552 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\") pod \"cf6e9741-0ee4-4956-9513-0b9552e6a867\" (UID: \"cf6e9741-0ee4-4956-9513-0b9552e6a867\") " Oct 30 20:57:03.815160 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.811997 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca" pod "cf6e9741-0ee4-4956-9513-0b9552e6a867" (UID: "cf6e9741-0ee4-4956-9513-0b9552e6a867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:03.841568 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.841502 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p" pod "cf6e9741-0ee4-4956-9513-0b9552e6a867" (UID: "cf6e9741-0ee4-4956-9513-0b9552e6a867"). InnerVolumeSpecName "kube-api-access-drm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:03.849687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.846880 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca" pod "cf6e9741-0ee4-4956-9513-0b9552e6a867" (UID: "cf6e9741-0ee4-4956-9513-0b9552e6a867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:03.921697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.917338 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drm9p\" (UniqueName: \"kubernetes.io/projected/cf6e9741-0ee4-4956-9513-0b9552e6a867-kube-api-access-drm9p\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:03.921697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.917380 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:03.921697 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:03.917397 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf6e9741-0ee4-4956-9513-0b9552e6a867-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:04.480919 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:04.480886 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" Oct 30 20:57:04.481103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:04.480894 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd" event={"ID":"cf6e9741-0ee4-4956-9513-0b9552e6a867","Type":"ContainerDied","Data":"ddc91e1191461fd297b13b4ec8ece4107c5c4e4d317b87fa9d9def26082b867e"} Oct 30 20:57:04.481103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:04.481020 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc91e1191461fd297b13b4ec8ece4107c5c4e4d317b87fa9d9def26082b867e" Oct 30 20:57:04.781730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:04.781633 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f02a5f-958a-4ab6-a452-3b4d06d440bd" path="/var/lib/kubelet/pods/82f02a5f-958a-4ab6-a452-3b4d06d440bd/volumes" Oct 30 20:57:04.782279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:04.782262 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f05e3f-15c9-49c0-bb76-5135ce9fbb81" path="/var/lib/kubelet/pods/a1f05e3f-15c9-49c0-bb76-5135ce9fbb81/volumes" Oct 30 20:57:09.819529 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:09.819470 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" containerID="cri-o://f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8" gracePeriod=45 Oct 30 20:57:09.856895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:09.856847 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" containerID="cri-o://eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48" gracePeriod=45 Oct 30 20:57:10.552753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.511873 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.829590 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.829684 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\" (UniqueName: \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.829729 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.829904 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.830130 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.830243 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\") pod \"7ddef63a-06bb-44b2-8503-0868fc054338\" (UID: \"7ddef63a-06bb-44b2-8503-0868fc054338\") " Oct 30 20:57:10.832728 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.831100 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:10.874880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.874838 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "kube-api-access-gdlpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:10.879860 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.879818 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:10.900250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.899881 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:10.900250 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.900169 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:10.956073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.955469 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ddef63a-06bb-44b2-8503-0868fc054338-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.956073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.955508 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlpg\" (UniqueName: \"kubernetes.io/projected/7ddef63a-06bb-44b2-8503-0868fc054338-kube-api-access-gdlpg\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.956073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.955525 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.956073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.955540 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.956073 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:10.955554 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.044295 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.044238 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm" pod "7ddef63a-06bb-44b2-8503-0868fc054338" (UID: "7ddef63a-06bb-44b2-8503-0868fc054338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:11.061729 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.053341 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:57:11.066300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.066238 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ddef63a-06bb-44b2-8503-0868fc054338-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.088786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.084031 5855 generic.go:356] "Generic (PLEG): container finished" podID="7ddef63a-06bb-44b2-8503-0868fc054338" containerID="f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8" exitCode=0 Oct 30 20:57:11.088786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.084323 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" event={"ID":"7ddef63a-06bb-44b2-8503-0868fc054338","Type":"ContainerDied","Data":"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8"} Oct 30 20:57:11.088786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.084423 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" Oct 30 20:57:11.090061 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.090001 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf" event={"ID":"7ddef63a-06bb-44b2-8503-0868fc054338","Type":"ContainerDied","Data":"24af095dd57e3572d1d76183ba6eb13d1e527ef215b60bfb2ddf9d298dd7e4ed"} Oct 30 20:57:11.090316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.090261 5855 scope.go:122] "RemoveContainer" containerID="f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8" Oct 30 20:57:11.122211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.121190 5855 generic.go:356] "Generic (PLEG): container finished" podID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerID="eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48" exitCode=0 Oct 30 20:57:11.122211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.121337 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" event={"ID":"4a3b9107-6606-4a8c-806f-da7e770d09ae","Type":"ContainerDied","Data":"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48"} Oct 30 20:57:11.122211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.121397 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" event={"ID":"4a3b9107-6606-4a8c-806f-da7e770d09ae","Type":"ContainerDied","Data":"36fb9e9b21ada0ba52e1b1fcce7e038446828d811aeb4202e5519cb5245e9962"} Oct 30 20:57:11.122211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.121493 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188332 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188398 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\" (UniqueName: \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188431 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188547 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188722 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.188830 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\") pod \"4a3b9107-6606-4a8c-806f-da7e770d09ae\" (UID: \"4a3b9107-6606-4a8c-806f-da7e770d09ae\") " Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.205710 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:11.208077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.205938 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:11.220698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.217548 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:11.296064 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.296025 5855 scope.go:122] "RemoveContainer" containerID="f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.309757 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.309800 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.309817 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:57:11.311436 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8\": container with ID starting with f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8 not found: ID does not exist" containerID="f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.311549 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8"} err="failed to get container status \"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8\": rpc error: code = NotFound desc = could not find container \"f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8\": container with ID starting with f57e1a59685aeb851c1fad812d45fe933e78b45c67b957f7471db2ac23abf6d8 not found: ID does not exist" Oct 30 20:57:11.311696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.311579 5855 scope.go:122] "RemoveContainer" containerID="eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48" Oct 30 20:57:11.349888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.328855 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "kube-api-access-f95gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:11.349888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.338492 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:11.364172 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.362768 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:11.373750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.365452 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs" pod "4a3b9107-6606-4a8c-806f-da7e770d09ae" (UID: "4a3b9107-6606-4a8c-806f-da7e770d09ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:11.377941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.376762 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:11.389792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.378239 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 30 20:57:11.389792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.378336 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 20:57:11.389792 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.387886 5855 scope.go:122] "RemoveContainer" containerID="eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48" Oct 30 20:57:11.395696 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:57:11.391225 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48\": container with ID starting with eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48 not found: ID does not exist" containerID="eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48" Oct 30 20:57:11.395696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.391267 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48"} err="failed to get container status \"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48\": rpc error: code = NotFound desc = could not find container \"eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48\": container with ID starting with eab47e538139f3f19716b196f14766299a6e15fc2bd1b40229e79a0a19a49a48 not found: ID does not exist" Oct 30 20:57:11.439530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.411131 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:57:11.439530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.411269 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a3b9107-6606-4a8c-806f-da7e770d09ae-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.439530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.411304 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a3b9107-6606-4a8c-806f-da7e770d09ae-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.439530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.433210 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f95gj\" (UniqueName: \"kubernetes.io/projected/4a3b9107-6606-4a8c-806f-da7e770d09ae-kube-api-access-f95gj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:11.440251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.439984 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:11.471173 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.470295 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/simulator-w2-db534b-kserve-6f5f788cb5-s5ftf"] Oct 30 20:57:11.524923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.524889 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:57:11.586434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.585897 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/premium-w2-db534b-kserve-5dc67b9fb8-dgfqn"] Oct 30 20:57:11.905841 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.905788 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:11.985681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:11.963825 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:12.783068 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:12.783027 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" path="/var/lib/kubelet/pods/4a3b9107-6606-4a8c-806f-da7e770d09ae/volumes" Oct 30 20:57:12.783786 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:12.783768 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" path="/var/lib/kubelet/pods/7ddef63a-06bb-44b2-8503-0868fc054338/volumes" Oct 30 20:57:13.525735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.525692 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n"] Oct 30 20:57:13.526304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526288 5855 cpu_manager.go:414] "containerMap: removing container" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" Oct 30 20:57:13.526304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526304 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526320 5855 cpu_manager.go:414] "containerMap: removing container" podUID="cf6e9741-0ee4-4956-9513-0b9552e6a867" containerName="revoke-keys" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526325 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6e9741-0ee4-4956-9513-0b9552e6a867" containerName="revoke-keys" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526350 5855 cpu_manager.go:414] "containerMap: removing container" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526355 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526362 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e15da1b9-a128-4942-a8c2-d33db7c92c1a" containerName="revoke-keys" Oct 30 20:57:13.526441 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526367 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15da1b9-a128-4942-a8c2-d33db7c92c1a" containerName="revoke-keys" Oct 30 20:57:13.526755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526456 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3b9107-6606-4a8c-806f-da7e770d09ae" containerName="main" Oct 30 20:57:13.526755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526463 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15da1b9-a128-4942-a8c2-d33db7c92c1a" containerName="revoke-keys" Oct 30 20:57:13.526755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526474 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddef63a-06bb-44b2-8503-0868fc054338" containerName="main" Oct 30 20:57:13.526755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.526480 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6e9741-0ee4-4956-9513-0b9552e6a867" containerName="revoke-keys" Oct 30 20:57:13.533195 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.532567 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk"] Oct 30 20:57:13.534471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.534443 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.539875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.539846 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n"] Oct 30 20:57:13.540178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.540161 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.541947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.541918 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:13.542094 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.542072 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:57:13.542225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.542143 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:13.542225 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.542207 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:57:13.547541 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.547504 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk"] Oct 30 20:57:13.626283 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626235 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.626481 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626292 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.626481 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626453 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrl6\" (UniqueName: \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.626559 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626507 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdnfh\" (UniqueName: \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.626698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626639 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.626735 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.626720 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.728894 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.728863 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.729086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.728905 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.729086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.728983 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrl6\" (UniqueName: \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.729086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.729029 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdnfh\" (UniqueName: \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.729189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.729137 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.729189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.729171 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.729829 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.729745 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.730015 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.729995 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.731801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.731772 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.731801 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.731789 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.742569 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.742538 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrl6\" (UniqueName: \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\") pod \"maas-api-revoke-sub-4b13184a-xtkgk\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:13.742931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.742907 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdnfh\" (UniqueName: \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\") pod \"maas-api-revoke-sub-fc264cc3-dsj6n\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.865269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.864931 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:13.867954 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:13.867922 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:14.046938 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:14.046895 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s"] Oct 30 20:57:14.053703 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:14.053669 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767543ca-jzn2s"] Oct 30 20:57:14.330814 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:14.330786 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n"] Oct 30 20:57:14.332552 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:57:14.332520 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafbb32f_9f7e_40ab_bd62_58b90f6c852f.slice/crio-c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701 WatchSource:0}: Error finding container c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701: Status 404 returned error can't find the container with id c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701 Oct 30 20:57:14.381483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:14.381413 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk"] Oct 30 20:57:14.956768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:14.954613 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae59a59e-0964-4bfa-8e6b-c498354d291a" path="/var/lib/kubelet/pods/ae59a59e-0964-4bfa-8e6b-c498354d291a/volumes" Oct 30 20:57:15.033428 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.033387 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9"] Oct 30 20:57:15.040098 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.040060 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99"] Oct 30 20:57:15.044805 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.044772 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d17cf35-28md9"] Oct 30 20:57:15.049268 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.049243 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd58f743-xjm99"] Oct 30 20:57:15.237712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.237591 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" event={"ID":"5dcdec09-632c-4c5f-9ba2-85de673420cc","Type":"ContainerStarted","Data":"43d6e47a3d9554c0a4f343a99d5ef6246d2142794049298b24cf939d6c909aa4"} Oct 30 20:57:15.237712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.237671 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" event={"ID":"5dcdec09-632c-4c5f-9ba2-85de673420cc","Type":"ContainerStarted","Data":"2e358fec5eb97cc1f24577688575a6b183bf19ac1cd462dc2d71689c98b8527b"} Oct 30 20:57:15.239339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.239307 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" event={"ID":"bafbb32f-9f7e-40ab-bd62-58b90f6c852f","Type":"ContainerStarted","Data":"a66e570ff4ad6d2cb831facd9325aec1c10774d89a7404b46b742e29d845e22e"} Oct 30 20:57:15.239467 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.239364 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" event={"ID":"bafbb32f-9f7e-40ab-bd62-58b90f6c852f","Type":"ContainerStarted","Data":"c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701"} Oct 30 20:57:15.258383 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.258331 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" podStartSLOduration=2.258316265 podStartE2EDuration="2.258316265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:15.256048023 +0000 UTC m=+1048.614760772" watchObservedRunningTime="2026-10-30 20:57:15.258316265 +0000 UTC m=+1048.617029011" Oct 30 20:57:15.285387 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:15.285322 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" podStartSLOduration=2.285299614 podStartE2EDuration="2.285299614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:15.27620863 +0000 UTC m=+1048.634921403" watchObservedRunningTime="2026-10-30 20:57:15.285299614 +0000 UTC m=+1048.644012362" Oct 30 20:57:16.320385 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.320346 5855 generic.go:356] "Generic (PLEG): container finished" podID="5dcdec09-632c-4c5f-9ba2-85de673420cc" containerID="43d6e47a3d9554c0a4f343a99d5ef6246d2142794049298b24cf939d6c909aa4" exitCode=0 Oct 30 20:57:16.321074 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.321045 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" event={"ID":"5dcdec09-632c-4c5f-9ba2-85de673420cc","Type":"ContainerDied","Data":"43d6e47a3d9554c0a4f343a99d5ef6246d2142794049298b24cf939d6c909aa4"} Oct 30 20:57:16.326579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.326543 5855 generic.go:356] "Generic (PLEG): container finished" podID="bafbb32f-9f7e-40ab-bd62-58b90f6c852f" containerID="a66e570ff4ad6d2cb831facd9325aec1c10774d89a7404b46b742e29d845e22e" exitCode=0 Oct 30 20:57:16.327034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.327009 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" event={"ID":"bafbb32f-9f7e-40ab-bd62-58b90f6c852f","Type":"ContainerDied","Data":"a66e570ff4ad6d2cb831facd9325aec1c10774d89a7404b46b742e29d845e22e"} Oct 30 20:57:16.783870 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.783781 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43027be5-25ba-492f-9824-3a9ae85dab5b" path="/var/lib/kubelet/pods/43027be5-25ba-492f-9824-3a9ae85dab5b/volumes" Oct 30 20:57:16.784427 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:16.784408 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621956fe-e412-4fe9-9844-bfd7993a31ca" path="/var/lib/kubelet/pods/621956fe-e412-4fe9-9844-bfd7993a31ca/volumes" Oct 30 20:57:17.047739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.044713 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk"] Oct 30 20:57:17.052766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.051377 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f649a51-6nmvk"] Oct 30 20:57:17.588770 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.588696 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:17.592287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.592263 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:17.752506 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752383 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\") pod \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " Oct 30 20:57:17.752776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752521 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\") pod \"5dcdec09-632c-4c5f-9ba2-85de673420cc\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " Oct 30 20:57:17.752776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752714 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\") pod \"5dcdec09-632c-4c5f-9ba2-85de673420cc\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " Oct 30 20:57:17.752917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752776 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\" (UniqueName: \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\") pod \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " Oct 30 20:57:17.752917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752820 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\" (UniqueName: \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\") pod \"5dcdec09-632c-4c5f-9ba2-85de673420cc\" (UID: \"5dcdec09-632c-4c5f-9ba2-85de673420cc\") " Oct 30 20:57:17.752917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.752893 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\") pod \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\" (UID: \"bafbb32f-9f7e-40ab-bd62-58b90f6c852f\") " Oct 30 20:57:17.808699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.792376 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca" pod "5dcdec09-632c-4c5f-9ba2-85de673420cc" (UID: "5dcdec09-632c-4c5f-9ba2-85de673420cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:17.820896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.820050 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca" pod "bafbb32f-9f7e-40ab-bd62-58b90f6c852f" (UID: "bafbb32f-9f7e-40ab-bd62-58b90f6c852f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:17.827741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.824849 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca" pod "5dcdec09-632c-4c5f-9ba2-85de673420cc" (UID: "5dcdec09-632c-4c5f-9ba2-85de673420cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:17.835910 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.835866 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca" pod "bafbb32f-9f7e-40ab-bd62-58b90f6c852f" (UID: "bafbb32f-9f7e-40ab-bd62-58b90f6c852f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:17.836116 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.836088 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6" pod "5dcdec09-632c-4c5f-9ba2-85de673420cc" (UID: "5dcdec09-632c-4c5f-9ba2-85de673420cc"). InnerVolumeSpecName "kube-api-access-rwrl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:17.844210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.844119 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh" pod "bafbb32f-9f7e-40ab-bd62-58b90f6c852f" (UID: "bafbb32f-9f7e-40ab-bd62-58b90f6c852f"). InnerVolumeSpecName "kube-api-access-bdnfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:17.861222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861181 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:17.861222 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861222 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:17.861485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861237 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dcdec09-632c-4c5f-9ba2-85de673420cc-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:17.861485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861268 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdnfh\" (UniqueName: \"kubernetes.io/projected/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-kube-api-access-bdnfh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:17.861485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861281 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrl6\" (UniqueName: \"kubernetes.io/projected/5dcdec09-632c-4c5f-9ba2-85de673420cc-kube-api-access-rwrl6\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:17.861485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:17.861295 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafbb32f-9f7e-40ab-bd62-58b90f6c852f-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:18.350999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.350959 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" event={"ID":"5dcdec09-632c-4c5f-9ba2-85de673420cc","Type":"ContainerDied","Data":"2e358fec5eb97cc1f24577688575a6b183bf19ac1cd462dc2d71689c98b8527b"} Oct 30 20:57:18.350999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.350987 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk" Oct 30 20:57:18.351252 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.351019 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e358fec5eb97cc1f24577688575a6b183bf19ac1cd462dc2d71689c98b8527b" Oct 30 20:57:18.352809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.352786 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" Oct 30 20:57:18.352914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.352814 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n" event={"ID":"bafbb32f-9f7e-40ab-bd62-58b90f6c852f","Type":"ContainerDied","Data":"c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701"} Oct 30 20:57:18.352914 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.352851 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17b641f1717a03773aa3fe2c09423067560d3cc3dfddfcad6728e014af7f701" Oct 30 20:57:18.781288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:18.781206 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a717127-40cf-49f8-8196-f6ab134233ba" path="/var/lib/kubelet/pods/1a717127-40cf-49f8-8196-f6ab134233ba/volumes" Oct 30 20:57:21.286380 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.286347 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r"] Oct 30 20:57:21.287193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287169 5855 cpu_manager.go:414] "containerMap: removing container" podUID="bafbb32f-9f7e-40ab-bd62-58b90f6c852f" containerName="revoke-keys" Oct 30 20:57:21.287290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287195 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafbb32f-9f7e-40ab-bd62-58b90f6c852f" containerName="revoke-keys" Oct 30 20:57:21.287290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287248 5855 cpu_manager.go:414] "containerMap: removing container" podUID="5dcdec09-632c-4c5f-9ba2-85de673420cc" containerName="revoke-keys" Oct 30 20:57:21.287290 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287256 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcdec09-632c-4c5f-9ba2-85de673420cc" containerName="revoke-keys" Oct 30 20:57:21.287447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287424 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcdec09-632c-4c5f-9ba2-85de673420cc" containerName="revoke-keys" Oct 30 20:57:21.287447 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.287444 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafbb32f-9f7e-40ab-bd62-58b90f6c852f" containerName="revoke-keys" Oct 30 20:57:21.292844 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.292820 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.309530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.309495 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:57:21.316738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.310560 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:57:21.316738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.310700 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:21.316738 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.310738 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:21.345970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.345932 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r"] Oct 30 20:57:21.374342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.374298 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjg8\" (UniqueName: \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.374563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.374356 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.374762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.374736 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.505061 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.505023 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.513430 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.513394 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.513968 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.513944 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjg8\" (UniqueName: \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.514133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.514118 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.556426 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.556365 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.557280 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.557255 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjg8\" (UniqueName: \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\") pod \"maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.652289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.652251 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:21.855937 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:21.855838 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:57:22.048679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:22.047176 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r"] Oct 30 20:57:22.536275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:22.536191 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" event={"ID":"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0","Type":"ContainerStarted","Data":"98c74ce3dae8049553725ef79a35de5565ed296bc00b1670034d59fe7550f45f"} Oct 30 20:57:23.555730 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:23.555670 5855 generic.go:356] "Generic (PLEG): container finished" podID="7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" containerID="2ca81756361fde404eaa27edcd21127711bfb03c0d03518cf36af6824222b5bf" exitCode=0 Oct 30 20:57:23.556393 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:23.556366 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" event={"ID":"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0","Type":"ContainerDied","Data":"2ca81756361fde404eaa27edcd21127711bfb03c0d03518cf36af6824222b5bf"} Oct 30 20:57:25.000221 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.000186 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:25.133485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.133442 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\" (UniqueName: \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\") pod \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " Oct 30 20:57:25.133850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.133828 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\") pod \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " Oct 30 20:57:25.134032 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.134015 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\") pod \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\" (UID: \"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0\") " Oct 30 20:57:25.135633 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.135602 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca" pod "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" (UID: "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:25.149261 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.149191 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8" pod "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" (UID: "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0"). InnerVolumeSpecName "kube-api-access-mwjg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:25.151934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.151895 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca" pod "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" (UID: "7ff9baf6-26c6-4fb5-ab46-4ae619b434e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:25.237218 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.237122 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjg8\" (UniqueName: \"kubernetes.io/projected/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-kube-api-access-mwjg8\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.237958 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.237933 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.238107 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.238094 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:25.689779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.689738 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" event={"ID":"7ff9baf6-26c6-4fb5-ab46-4ae619b434e0","Type":"ContainerDied","Data":"98c74ce3dae8049553725ef79a35de5565ed296bc00b1670034d59fe7550f45f"} Oct 30 20:57:25.690070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.690051 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c74ce3dae8049553725ef79a35de5565ed296bc00b1670034d59fe7550f45f" Oct 30 20:57:25.690260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:25.690246 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r" Oct 30 20:57:26.049747 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:26.046712 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq"] Oct 30 20:57:26.065016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:26.064964 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aad23c4-ldrqq"] Oct 30 20:57:26.835540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:26.835497 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b067a56d-f9ed-4f57-a6d8-fd059bd6ea45" path="/var/lib/kubelet/pods/b067a56d-f9ed-4f57-a6d8-fd059bd6ea45/volumes" Oct 30 20:57:28.087085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.087046 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw"] Oct 30 20:57:28.092857 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.092828 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223d4523-ckrzw"] Oct 30 20:57:28.645616 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.645573 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7"] Oct 30 20:57:28.646379 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.646359 5855 cpu_manager.go:414] "containerMap: removing container" podUID="7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" containerName="revoke-keys" Oct 30 20:57:28.646501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.646379 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" containerName="revoke-keys" Oct 30 20:57:28.646571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.646535 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" containerName="revoke-keys" Oct 30 20:57:28.651923 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.651899 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.658114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.658085 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7"] Oct 30 20:57:28.658270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.658211 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:57:28.658270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.658211 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:57:28.658270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.658230 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:28.658401 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.658278 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:28.714179 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.714137 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.714574 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.714548 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xd8\" (UniqueName: \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.714843 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.714823 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.782439 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.782392 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b11dca7-689a-44ab-8744-91046600e2b0" path="/var/lib/kubelet/pods/8b11dca7-689a-44ab-8744-91046600e2b0/volumes" Oct 30 20:57:28.816612 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.816567 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.817062 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.817035 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xd8\" (UniqueName: \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.817312 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.817287 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.818167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.818137 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.819488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.819465 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.831038 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.830980 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xd8\" (UniqueName: \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\") pod \"maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:28.970732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:28.970617 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:29.477680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:29.476833 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd"] Oct 30 20:57:29.549582 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:29.549070 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-db534b-r8cqd"] Oct 30 20:57:29.807418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:29.807382 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7"] Oct 30 20:57:29.852479 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:57:29.852442 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cd2d01_4a3e_4167_b8c1_0915b45decf2.slice/crio-6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2 WatchSource:0}: Error finding container 6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2: Status 404 returned error can't find the container with id 6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2 Oct 30 20:57:30.026997 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.026945 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj"] Oct 30 20:57:30.032089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.032060 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5944c6c5-bppqj"] Oct 30 20:57:30.783979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.783948 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6e9741-0ee4-4956-9513-0b9552e6a867" path="/var/lib/kubelet/pods/cf6e9741-0ee4-4956-9513-0b9552e6a867/volumes" Oct 30 20:57:30.784463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.784448 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30dc773-0210-4fc6-baed-bbe38f88cc9b" path="/var/lib/kubelet/pods/f30dc773-0210-4fc6-baed-bbe38f88cc9b/volumes" Oct 30 20:57:30.833528 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.833497 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" event={"ID":"e4cd2d01-4a3e-4167-b8c1-0915b45decf2","Type":"ContainerStarted","Data":"7458b56b615022431eeccd814a19de2a9dd758773c4b9d263fb5e86ced383762"} Oct 30 20:57:30.833745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:30.833555 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" event={"ID":"e4cd2d01-4a3e-4167-b8c1-0915b45decf2","Type":"ContainerStarted","Data":"6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2"} Oct 30 20:57:30.968508 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:57:30.968461 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:57:31.900771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:31.900017 5855 generic.go:356] "Generic (PLEG): container finished" podID="e4cd2d01-4a3e-4167-b8c1-0915b45decf2" containerID="7458b56b615022431eeccd814a19de2a9dd758773c4b9d263fb5e86ced383762" exitCode=0 Oct 30 20:57:31.900771 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:31.900231 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" event={"ID":"e4cd2d01-4a3e-4167-b8c1-0915b45decf2","Type":"ContainerDied","Data":"7458b56b615022431eeccd814a19de2a9dd758773c4b9d263fb5e86ced383762"} Oct 30 20:57:34.119085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.119004 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:34.236748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.234771 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\") pod \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " Oct 30 20:57:34.236748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.234882 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\" (UniqueName: \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\") pod \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " Oct 30 20:57:34.236748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.234930 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\") pod \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\" (UID: \"e4cd2d01-4a3e-4167-b8c1-0915b45decf2\") " Oct 30 20:57:34.236748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.235478 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca" pod "e4cd2d01-4a3e-4167-b8c1-0915b45decf2" (UID: "e4cd2d01-4a3e-4167-b8c1-0915b45decf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:34.236748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.236260 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:34.238389 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.238357 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8" pod "e4cd2d01-4a3e-4167-b8c1-0915b45decf2" (UID: "e4cd2d01-4a3e-4167-b8c1-0915b45decf2"). InnerVolumeSpecName "kube-api-access-j5xd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:34.240552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.240521 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca" pod "e4cd2d01-4a3e-4167-b8c1-0915b45decf2" (UID: "e4cd2d01-4a3e-4167-b8c1-0915b45decf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:34.337725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.337688 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5xd8\" (UniqueName: \"kubernetes.io/projected/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-kube-api-access-j5xd8\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:34.337725 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:34.337728 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4cd2d01-4a3e-4167-b8c1-0915b45decf2-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:35.025281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:35.025246 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" Oct 30 20:57:35.025281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:35.025257 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7" event={"ID":"e4cd2d01-4a3e-4167-b8c1-0915b45decf2","Type":"ContainerDied","Data":"6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2"} Oct 30 20:57:35.025572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:35.025351 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6293e2ff8ed0d34ab811ca7422c046564ec168c9f371c4bfb48e55f778aaa7c2" Oct 30 20:57:36.025777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:36.025731 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5"] Oct 30 20:57:36.034137 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:36.034108 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f47d1e7a-gxtc5"] Oct 30 20:57:36.783545 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:36.783509 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd659143-bb47-4ba6-9842-d35c9b294c05" path="/var/lib/kubelet/pods/dd659143-bb47-4ba6-9842-d35c9b294c05/volumes" Oct 30 20:57:37.410054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:37.409828 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r"] Oct 30 20:57:37.558736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:37.556084 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a4317e67-l9m8r"] Oct 30 20:57:38.133877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.133845 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:57:38.134395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.134364 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" containerID="cri-o://f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1" gracePeriod=30 Oct 30 20:57:38.232304 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.223457 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:57:38.236679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.236592 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" containerID="cri-o://366cc0c898487958188a5252c4f9ab368d4fb7a8ab3c863bc3650c92a9140872" gracePeriod=30 Oct 30 20:57:38.298783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.293955 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:57:38.298783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.294206 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" containerID="cri-o://1d39054ff89bc70ae381587387ca268fbb6d0ef52d45d1937e87fc007f9eb17b" gracePeriod=30 Oct 30 20:57:38.708207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.706908 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 20:57:38.958987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.948794 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 20:57:38.975289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:38.975221 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 20:57:39.057707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.054237 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:39.265981 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.253022 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:39.452949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.452879 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:39.479948 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.479900 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:39.506573 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.506533 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff9baf6-26c6-4fb5-ab46-4ae619b434e0" path="/var/lib/kubelet/pods/7ff9baf6-26c6-4fb5-ab46-4ae619b434e0/volumes" Oct 30 20:57:39.570443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.513945 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 20:57:39.570443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.559302 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv"] Oct 30 20:57:39.570443 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.563825 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5847cfe-smwkv"] Oct 30 20:57:39.923752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.907088 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:39.923752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:39.907150 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:40.788853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:40.788816 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ee7f26-e162-4b4d-8bcc-e8755dd957bc" path="/var/lib/kubelet/pods/98ee7f26-e162-4b4d-8bcc-e8755dd957bc/volumes" Oct 30 20:57:41.404634 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:41.404568 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:41.405155 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:41.404643 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:41.703973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:41.701702 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7"] Oct 30 20:57:41.739687 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:41.738900 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8d0462d9-gxjb7"] Oct 30 20:57:42.489459 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.489416 5855 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:42.489917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.489496 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:42.561987 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.561953 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg"] Oct 30 20:57:42.562822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.562800 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e4cd2d01-4a3e-4167-b8c1-0915b45decf2" containerName="revoke-keys" Oct 30 20:57:42.562822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.562820 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cd2d01-4a3e-4167-b8c1-0915b45decf2" containerName="revoke-keys" Oct 30 20:57:42.563054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.562983 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cd2d01-4a3e-4167-b8c1-0915b45decf2" containerName="revoke-keys" Oct 30 20:57:42.568211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.568188 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.573176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.573142 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:42.573852 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.573739 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:57:42.574091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.573988 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:42.574091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.574064 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:57:42.574732 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.574709 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg"] Oct 30 20:57:42.679758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.678497 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.679758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.679044 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7545r\" (UniqueName: \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.679758 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.679120 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.782049 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.781919 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cd2d01-4a3e-4167-b8c1-0915b45decf2" path="/var/lib/kubelet/pods/e4cd2d01-4a3e-4167-b8c1-0915b45decf2/volumes" Oct 30 20:57:42.782353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.782152 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.782483 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.782456 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.782704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.782635 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7545r\" (UniqueName: \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.783341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.783315 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.785597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.785559 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.795578 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.795542 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7545r\" (UniqueName: \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\") pod \"maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:42.996611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:42.996573 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:43.374856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.374763 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg"] Oct 30 20:57:43.418525 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:57:43.418446 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83a039c_0218_43be_902e_6f5aeb063647.slice/crio-7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616 WatchSource:0}: Error finding container 7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616: Status 404 returned error can't find the container with id 7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616 Oct 30 20:57:43.645779 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.645749 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:57:43.703279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703241 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703437 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703324 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703571 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703550 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703681 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703745 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703754 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.703897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703802 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.703897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703837 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.706787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703902 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.706787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.703996 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\") pod \"2e8bc822-203d-488f-84f0-0595ebcabe19\" (UID: \"2e8bc822-203d-488f-84f0-0595ebcabe19\") " Oct 30 20:57:43.706787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.704331 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.706787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.704707 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:43.706787 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.705052 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.707495 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.707443 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.708291 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708263 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "kube-api-access-9m5vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.708890 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708864 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5vj\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-kube-api-access-9m5vj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.709003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708893 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.709003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708909 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.709003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708925 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8bc822-203d-488f-84f0-0595ebcabe19-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.709003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708938 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.709003 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.708951 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.716315 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.716263 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:43.728308 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.728274 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.729000 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.728975 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token" pod "2e8bc822-203d-488f-84f0-0595ebcabe19" (UID: "2e8bc822-203d-488f-84f0-0595ebcabe19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.774734 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.774697 5855 generic.go:356] "Generic (PLEG): container finished" podID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerID="1d39054ff89bc70ae381587387ca268fbb6d0ef52d45d1937e87fc007f9eb17b" exitCode=0 Oct 30 20:57:43.774908 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.774745 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" event={"ID":"3f5af8da-cf9f-440d-b02a-6a473bac051a","Type":"ContainerDied","Data":"1d39054ff89bc70ae381587387ca268fbb6d0ef52d45d1937e87fc007f9eb17b"} Oct 30 20:57:43.776861 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.776817 5855 generic.go:356] "Generic (PLEG): container finished" podID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerID="f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1" exitCode=0 Oct 30 20:57:43.777084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.776995 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" Oct 30 20:57:43.777084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.777011 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" event={"ID":"2e8bc822-203d-488f-84f0-0595ebcabe19","Type":"ContainerDied","Data":"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1"} Oct 30 20:57:43.777084 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.777072 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd" event={"ID":"2e8bc822-203d-488f-84f0-0595ebcabe19","Type":"ContainerDied","Data":"99cb61288e61eb758d84e59052cf40ce3c6f70fd3765282ecc2e19140578f1bc"} Oct 30 20:57:43.777236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.777103 5855 scope.go:122] "RemoveContainer" containerID="f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1" Oct 30 20:57:43.780104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.780062 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" event={"ID":"d83a039c-0218-43be-902e-6f5aeb063647","Type":"ContainerStarted","Data":"7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616"} Oct 30 20:57:43.782638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.782415 5855 generic.go:356] "Generic (PLEG): container finished" podID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerID="366cc0c898487958188a5252c4f9ab368d4fb7a8ab3c863bc3650c92a9140872" exitCode=0 Oct 30 20:57:43.782638 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.782521 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" event={"ID":"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e","Type":"ContainerDied","Data":"366cc0c898487958188a5252c4f9ab368d4fb7a8ab3c863bc3650c92a9140872"} Oct 30 20:57:43.862962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.831348 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.862962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.831383 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.862962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.831401 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8bc822-203d-488f-84f0-0595ebcabe19-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.928989 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.928913 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:57:43.935929 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.935912 5855 scope.go:122] "RemoveContainer" containerID="f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1" Oct 30 20:57:43.936607 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:57:43.936584 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1\": container with ID starting with f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1 not found: ID does not exist" containerID="f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1" Oct 30 20:57:43.936956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.936812 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1"} err="failed to get container status \"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1\": rpc error: code = NotFound desc = could not find container \"f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1\": container with ID starting with f9e20ad6df16d848195c8d17c0f9684fad97021d83bcf27c8df9be2e86eae8f1 not found: ID does not exist" Oct 30 20:57:43.942493 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.942468 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:57:43.948067 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.948032 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:57:43.952777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:43.952750 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5xxqd"] Oct 30 20:57:44.042089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042037 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042126 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042173 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042229 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042276 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042262 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042294 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042361 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042412 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042431 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042487 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.042520 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042512 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042540 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042552 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042580 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042722 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042726 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042832 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.042865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042858 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042880 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.042896 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043028 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\") pod \"3f5af8da-cf9f-440d-b02a-6a473bac051a\" (UID: \"3f5af8da-cf9f-440d-b02a-6a473bac051a\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043065 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043111 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043155 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\") pod \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\" (UID: \"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e\") " Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043192 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.043229 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043217 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.043717 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043590 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.044056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.043959 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:44.044147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044086 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044107 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044124 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044147 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044139 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044154 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044169 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044183 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044196 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5af8da-cf9f-440d-b02a-6a473bac051a-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.044433 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.044210 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.046125 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.046094 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.047130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047095 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:44.047233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047198 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:44.047645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047620 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "kube-api-access-7l6fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:44.047645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047623 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:44.047837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047741 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "kube-api-access-jcgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:44.047933 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.047904 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy" pod "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" (UID: "4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:44.048093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.048073 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo" pod "3f5af8da-cf9f-440d-b02a-6a473bac051a" (UID: "3f5af8da-cf9f-440d-b02a-6a473bac051a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145238 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145266 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145277 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145286 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145296 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgzp\" (UniqueName: \"kubernetes.io/projected/3f5af8da-cf9f-440d-b02a-6a473bac051a-kube-api-access-jcgzp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145305 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145314 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145322 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145322 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5af8da-cf9f-440d-b02a-6a473bac051a-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.145720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.145332 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6fn\" (UniqueName: \"kubernetes.io/projected/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e-kube-api-access-7l6fn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:44.782078 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.782036 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" path="/var/lib/kubelet/pods/2e8bc822-203d-488f-84f0-0595ebcabe19/volumes" Oct 30 20:57:44.794476 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.794444 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" Oct 30 20:57:44.794600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.794468 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9" event={"ID":"3f5af8da-cf9f-440d-b02a-6a473bac051a","Type":"ContainerDied","Data":"3a6d43f73711c4df71124206d62177b1d4c4a818b1bde8f2bd45a9097d20b4e2"} Oct 30 20:57:44.794600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.794537 5855 scope.go:122] "RemoveContainer" containerID="1d39054ff89bc70ae381587387ca268fbb6d0ef52d45d1937e87fc007f9eb17b" Oct 30 20:57:44.798485 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.798462 5855 generic.go:356] "Generic (PLEG): container finished" podID="d83a039c-0218-43be-902e-6f5aeb063647" containerID="ff55308c78fd244f3e14ed628413dad9d10c8f95853e9e7745f22b9a799e7f87" exitCode=0 Oct 30 20:57:44.798581 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.798526 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" event={"ID":"d83a039c-0218-43be-902e-6f5aeb063647","Type":"ContainerDied","Data":"ff55308c78fd244f3e14ed628413dad9d10c8f95853e9e7745f22b9a799e7f87"} Oct 30 20:57:44.800327 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.800305 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" Oct 30 20:57:44.800414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.800324 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf" event={"ID":"4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e","Type":"ContainerDied","Data":"e940ef087af90775357da78202ed37489abbb2d0ebf1ced931ac5941b2d35ff3"} Oct 30 20:57:44.810438 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.810416 5855 scope.go:122] "RemoveContainer" containerID="366cc0c898487958188a5252c4f9ab368d4fb7a8ab3c863bc3650c92a9140872" Oct 30 20:57:44.818395 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.818367 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:57:44.823043 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.823015 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-d5js9"] Oct 30 20:57:44.852883 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.852850 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:57:44.858236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:44.858207 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-rktdf"] Oct 30 20:57:46.100973 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.100937 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:46.151213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.151172 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7"] Oct 30 20:57:46.152231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152207 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" Oct 30 20:57:46.152231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152232 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" Oct 30 20:57:46.152444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152343 5855 cpu_manager.go:414] "containerMap: removing container" podUID="d83a039c-0218-43be-902e-6f5aeb063647" containerName="revoke-keys" Oct 30 20:57:46.152444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152354 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83a039c-0218-43be-902e-6f5aeb063647" containerName="revoke-keys" Oct 30 20:57:46.152540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152366 5855 cpu_manager.go:414] "containerMap: removing container" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" Oct 30 20:57:46.152540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152501 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" Oct 30 20:57:46.152540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152515 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" Oct 30 20:57:46.152540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152523 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" Oct 30 20:57:46.152776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152728 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" containerName="istio-proxy" Oct 30 20:57:46.152776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152747 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83a039c-0218-43be-902e-6f5aeb063647" containerName="revoke-keys" Oct 30 20:57:46.152776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152757 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" containerName="istio-proxy" Oct 30 20:57:46.152776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.152768 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8bc822-203d-488f-84f0-0595ebcabe19" containerName="istio-proxy" Oct 30 20:57:46.157766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.157741 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.172169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.172090 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7"] Oct 30 20:57:46.204739 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.204649 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\" (UniqueName: \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\") pod \"d83a039c-0218-43be-902e-6f5aeb063647\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " Oct 30 20:57:46.204955 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.204910 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\") pod \"d83a039c-0218-43be-902e-6f5aeb063647\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " Oct 30 20:57:46.205039 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.204957 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\") pod \"d83a039c-0218-43be-902e-6f5aeb063647\" (UID: \"d83a039c-0218-43be-902e-6f5aeb063647\") " Oct 30 20:57:46.208970 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.208934 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca" pod "d83a039c-0218-43be-902e-6f5aeb063647" (UID: "d83a039c-0218-43be-902e-6f5aeb063647"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:46.209339 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.209311 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca" pod "d83a039c-0218-43be-902e-6f5aeb063647" (UID: "d83a039c-0218-43be-902e-6f5aeb063647"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:46.209859 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.209826 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r" pod "d83a039c-0218-43be-902e-6f5aeb063647" (UID: "d83a039c-0218-43be-902e-6f5aeb063647"). InnerVolumeSpecName "kube-api-access-7545r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:46.307455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307419 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.307645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307574 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4k4q\" (UniqueName: \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.307645 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307601 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.307748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307691 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:46.307794 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307773 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83a039c-0218-43be-902e-6f5aeb063647-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:46.307834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.307800 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7545r\" (UniqueName: \"kubernetes.io/projected/d83a039c-0218-43be-902e-6f5aeb063647-kube-api-access-7545r\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:46.408969 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.408864 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4k4q\" (UniqueName: \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.408969 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.408913 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.409218 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.409034 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.409815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.409792 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.411502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.411479 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.421819 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.421792 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4k4q\" (UniqueName: \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\") pod \"maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.489089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.489044 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:46.858621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.858591 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" Oct 30 20:57:46.877034 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.877000 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5af8da-cf9f-440d-b02a-6a473bac051a" path="/var/lib/kubelet/pods/3f5af8da-cf9f-440d-b02a-6a473bac051a/volumes" Oct 30 20:57:46.911482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.911446 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e" path="/var/lib/kubelet/pods/4c96e45a-2a89-44b8-84bd-1fd0cddf5e5e/volumes" Oct 30 20:57:46.944051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.944022 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7"] Oct 30 20:57:46.944187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.944108 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg" event={"ID":"d83a039c-0218-43be-902e-6f5aeb063647","Type":"ContainerDied","Data":"7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616"} Oct 30 20:57:46.944187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.944147 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7343f4d17a021790c73958258d75568313e15edc6beafd0853751be75d0af616" Oct 30 20:57:46.944187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:46.944177 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" event={"ID":"20120e07-b1c1-4dc9-a3cc-03be73dd73dc","Type":"ContainerStarted","Data":"c3d27ae0a336950cdf1cd7da10700abce21458fd30754205f7da6e20c5088201"} Oct 30 20:57:47.366373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:47.366334 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:57:47.379360 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:47.379322 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:57:47.396262 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:47.396220 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:57:47.866396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:47.866357 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" event={"ID":"20120e07-b1c1-4dc9-a3cc-03be73dd73dc","Type":"ContainerStarted","Data":"527f7d419d4348e1328c49ee6780c3b61857931720ac31bcdb222081d411de99"} Oct 30 20:57:47.889498 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:47.889436 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" podStartSLOduration=1.8894216350000002 podStartE2EDuration="1.889421635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:47.88577066 +0000 UTC m=+1081.244483405" watchObservedRunningTime="2026-10-30 20:57:47.889421635 +0000 UTC m=+1081.248134384" Oct 30 20:57:48.931184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:48.926878 5855 generic.go:356] "Generic (PLEG): container finished" podID="20120e07-b1c1-4dc9-a3cc-03be73dd73dc" containerID="527f7d419d4348e1328c49ee6780c3b61857931720ac31bcdb222081d411de99" exitCode=0 Oct 30 20:57:48.931982 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:48.930944 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" event={"ID":"20120e07-b1c1-4dc9-a3cc-03be73dd73dc","Type":"ContainerDied","Data":"527f7d419d4348e1328c49ee6780c3b61857931720ac31bcdb222081d411de99"} Oct 30 20:57:48.970665 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:48.970630 5855 scope.go:122] "RemoveContainer" containerID="1126bfb9b142589299b991b4c9069a4fc54e943a1fdf3838d4391ce4d6482dfe" Oct 30 20:57:48.985297 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:48.985268 5855 scope.go:122] "RemoveContainer" containerID="977ede348b53c531b894c0c7150ba62e82e258f1a00fe27ea88620e2bbe8ce8b" Oct 30 20:57:49.001130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:49.001106 5855 scope.go:122] "RemoveContainer" containerID="a53806d852cead9fb76f3b520f7cdd187befa0352d7e253c6d30bb47c0cdd792" Oct 30 20:57:49.013721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:49.013696 5855 scope.go:122] "RemoveContainer" containerID="ed797f2c9ec10716f5516770eabdb32b733c08ff4168cf04c7c54b91a7f4706f" Oct 30 20:57:50.034686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.034241 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz"] Oct 30 20:57:50.041874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.041720 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aac5e068-lkvjz"] Oct 30 20:57:50.128597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.128566 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:50.203842 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.203740 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\" (UniqueName: \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\") pod \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " Oct 30 20:57:50.204014 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.203987 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\") pod \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " Oct 30 20:57:50.204079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.204027 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\") pod \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\" (UID: \"20120e07-b1c1-4dc9-a3cc-03be73dd73dc\") " Oct 30 20:57:50.204836 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.204801 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca" pod "20120e07-b1c1-4dc9-a3cc-03be73dd73dc" (UID: "20120e07-b1c1-4dc9-a3cc-03be73dd73dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:50.239992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.232012 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca" pod "20120e07-b1c1-4dc9-a3cc-03be73dd73dc" (UID: "20120e07-b1c1-4dc9-a3cc-03be73dd73dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:50.239992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.232085 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q" pod "20120e07-b1c1-4dc9-a3cc-03be73dd73dc" (UID: "20120e07-b1c1-4dc9-a3cc-03be73dd73dc"). InnerVolumeSpecName "kube-api-access-s4k4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:50.325839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.307636 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4k4q\" (UniqueName: \"kubernetes.io/projected/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-kube-api-access-s4k4q\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:50.325839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.307702 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:50.325839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.307719 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20120e07-b1c1-4dc9-a3cc-03be73dd73dc-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:50.781382 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:50.781350 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6dc8361-83be-4a84-addd-edc151cf0cbb" path="/var/lib/kubelet/pods/e6dc8361-83be-4a84-addd-edc151cf0cbb/volumes" Oct 30 20:57:51.016690 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.014730 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:57:51.020679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.020348 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" event={"ID":"20120e07-b1c1-4dc9-a3cc-03be73dd73dc","Type":"ContainerDied","Data":"c3d27ae0a336950cdf1cd7da10700abce21458fd30754205f7da6e20c5088201"} Oct 30 20:57:51.020679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.020431 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3d27ae0a336950cdf1cd7da10700abce21458fd30754205f7da6e20c5088201" Oct 30 20:57:51.020679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.020517 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7" Oct 30 20:57:51.031931 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.030878 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:57:51.048584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.048550 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl"] Oct 30 20:57:51.061294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:51.061262 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e8f8ef-pxnxl"] Oct 30 20:57:52.782805 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:52.782767 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df46cc43-5151-48db-a5b2-f908bd89f16a" path="/var/lib/kubelet/pods/df46cc43-5151-48db-a5b2-f908bd89f16a/volumes" Oct 30 20:57:55.033526 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.033440 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh"] Oct 30 20:57:55.034325 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.034303 5855 cpu_manager.go:414] "containerMap: removing container" podUID="20120e07-b1c1-4dc9-a3cc-03be73dd73dc" containerName="revoke-keys" Oct 30 20:57:55.034424 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.034328 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20120e07-b1c1-4dc9-a3cc-03be73dd73dc" containerName="revoke-keys" Oct 30 20:57:55.034489 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.034476 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="20120e07-b1c1-4dc9-a3cc-03be73dd73dc" containerName="revoke-keys" Oct 30 20:57:55.040227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.039857 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.044761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.044734 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:55.044936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.044738 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:57:55.044936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.044738 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:57:55.045864 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.045844 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:55.047320 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.047295 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg"] Oct 30 20:57:55.078927 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.075010 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.080338 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.074912 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh"] Oct 30 20:57:55.080568 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.080553 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg"] Oct 30 20:57:55.137884 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.137846 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.138085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.137988 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.138128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.138119 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rgh\" (UniqueName: \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.239629 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.239594 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.239826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.239638 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rgh\" (UniqueName: \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.239880 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.239821 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.239941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.239925 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.240036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.240022 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.240089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.240077 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8v9x\" (UniqueName: \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.240745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.240717 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.242462 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.242442 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.252669 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.252632 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rgh\" (UniqueName: \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\") pod \"maas-api-revoke-sub-a121ae6d-dpkkh\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.341740 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.341699 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8v9x\" (UniqueName: \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.341953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.341827 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.342027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.341952 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.342522 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.342493 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.344560 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.344532 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.354824 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.354788 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8v9x\" (UniqueName: \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\") pod \"maas-api-revoke-sub-8bd1c62c-h7kdg\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.361597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.361559 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:55.420484 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.408948 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:55.816577 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:57:55.815469 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80ded81_5681_4a8c_a20a_d45a383d423b.slice/crio-c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d WatchSource:0}: Error finding container c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d: Status 404 returned error can't find the container with id c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d Oct 30 20:57:55.817463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.817403 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh"] Oct 30 20:57:55.823784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:55.823751 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg"] Oct 30 20:57:56.130227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:56.130131 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" event={"ID":"d80ded81-5681-4a8c-a20a-d45a383d423b","Type":"ContainerStarted","Data":"c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d"} Oct 30 20:57:56.132920 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:56.132892 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" event={"ID":"543c1b58-56e7-4c66-a866-4e132fc24c08","Type":"ContainerStarted","Data":"5ac596eb7ffaa3863ad7cefa62ec6c76133e8a8c8ff9b82d3a89bce3000bf6fa"} Oct 30 20:57:56.733678 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:56.732607 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:57:57.144198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.144153 5855 generic.go:356] "Generic (PLEG): container finished" podID="d80ded81-5681-4a8c-a20a-d45a383d423b" containerID="1c1a910469b562ecc37a80c3a53869319893097b79c8449aa3ee26e09b1a43cb" exitCode=0 Oct 30 20:57:57.144641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.144216 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" event={"ID":"d80ded81-5681-4a8c-a20a-d45a383d423b","Type":"ContainerDied","Data":"1c1a910469b562ecc37a80c3a53869319893097b79c8449aa3ee26e09b1a43cb"} Oct 30 20:57:57.146323 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.146290 5855 generic.go:356] "Generic (PLEG): container finished" podID="543c1b58-56e7-4c66-a866-4e132fc24c08" containerID="1bb82061c900d652e937a971b23ce50b75aa16b961c8d4a864128a7c9cf65962" exitCode=0 Oct 30 20:57:57.146470 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.146373 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" event={"ID":"543c1b58-56e7-4c66-a866-4e132fc24c08","Type":"ContainerDied","Data":"1bb82061c900d652e937a971b23ce50b75aa16b961c8d4a864128a7c9cf65962"} Oct 30 20:57:57.220500 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.220459 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl"] Oct 30 20:57:57.236753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.235509 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.236753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.236639 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl"] Oct 30 20:57:57.264429 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.264395 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.264619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.264491 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brvz\" (UniqueName: \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.264619 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.264532 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.366313 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.366280 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.366623 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.366585 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brvz\" (UniqueName: \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.366767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.366650 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.368024 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.367994 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.370488 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.370461 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.386551 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.386520 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brvz\" (UniqueName: \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\") pod \"maas-api-revoke-sub-fe5ffbca-wsxcl\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.556977 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.556933 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:57:57.937332 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:57:57.937298 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe019915_e577_45b7_96ae_ece4e2c1718b.slice/crio-f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145 WatchSource:0}: Error finding container f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145: Status 404 returned error can't find the container with id f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145 Oct 30 20:57:57.938694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:57.938625 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl"] Oct 30 20:57:58.158045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.158003 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" event={"ID":"fe019915-e577-45b7-96ae-ece4e2c1718b","Type":"ContainerStarted","Data":"f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145"} Oct 30 20:57:58.342457 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.342427 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:58.388564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.386267 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\") pod \"543c1b58-56e7-4c66-a866-4e132fc24c08\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " Oct 30 20:57:58.388564 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.386383 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\") pod \"543c1b58-56e7-4c66-a866-4e132fc24c08\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " Oct 30 20:57:58.471502 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.465389 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca" pod "543c1b58-56e7-4c66-a866-4e132fc24c08" (UID: "543c1b58-56e7-4c66-a866-4e132fc24c08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:58.553790 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.547425 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\" (UniqueName: \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\") pod \"543c1b58-56e7-4c66-a866-4e132fc24c08\" (UID: \"543c1b58-56e7-4c66-a866-4e132fc24c08\") " Oct 30 20:57:58.564646 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.564604 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca" pod "543c1b58-56e7-4c66-a866-4e132fc24c08" (UID: "543c1b58-56e7-4c66-a866-4e132fc24c08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.565896 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.565864 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.565953 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.565914 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/543c1b58-56e7-4c66-a866-4e132fc24c08-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.572203 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.571542 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x" pod "543c1b58-56e7-4c66-a866-4e132fc24c08" (UID: "543c1b58-56e7-4c66-a866-4e132fc24c08"). InnerVolumeSpecName "kube-api-access-w8v9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:58.591182 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.590988 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:58.667360 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.667322 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8v9x\" (UniqueName: \"kubernetes.io/projected/543c1b58-56e7-4c66-a866-4e132fc24c08-kube-api-access-w8v9x\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.768235 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.768130 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\" (UniqueName: \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\") pod \"d80ded81-5681-4a8c-a20a-d45a383d423b\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " Oct 30 20:57:58.768418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.768272 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\") pod \"d80ded81-5681-4a8c-a20a-d45a383d423b\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " Oct 30 20:57:58.768490 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.768429 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\") pod \"d80ded81-5681-4a8c-a20a-d45a383d423b\" (UID: \"d80ded81-5681-4a8c-a20a-d45a383d423b\") " Oct 30 20:57:58.768877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.768849 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca" pod "d80ded81-5681-4a8c-a20a-d45a383d423b" (UID: "d80ded81-5681-4a8c-a20a-d45a383d423b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:58.769373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.769347 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.770777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.770750 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca" pod "d80ded81-5681-4a8c-a20a-d45a383d423b" (UID: "d80ded81-5681-4a8c-a20a-d45a383d423b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:58.770873 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.770774 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh" pod "d80ded81-5681-4a8c-a20a-d45a383d423b" (UID: "d80ded81-5681-4a8c-a20a-d45a383d423b"). InnerVolumeSpecName "kube-api-access-l8rgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:58.870418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.870381 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8rgh\" (UniqueName: \"kubernetes.io/projected/d80ded81-5681-4a8c-a20a-d45a383d423b-kube-api-access-l8rgh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:58.870418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:58.870418 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80ded81-5681-4a8c-a20a-d45a383d423b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.167748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.167640 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" Oct 30 20:57:59.167748 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.167727 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg" event={"ID":"543c1b58-56e7-4c66-a866-4e132fc24c08","Type":"ContainerDied","Data":"5ac596eb7ffaa3863ad7cefa62ec6c76133e8a8c8ff9b82d3a89bce3000bf6fa"} Oct 30 20:57:59.168274 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.167799 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ac596eb7ffaa3863ad7cefa62ec6c76133e8a8c8ff9b82d3a89bce3000bf6fa" Oct 30 20:57:59.170408 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.170334 5855 generic.go:356] "Generic (PLEG): container finished" podID="fe019915-e577-45b7-96ae-ece4e2c1718b" containerID="5fad1dfd2189542806e4042504ee1037bf888193bc76287248d182a304e95919" exitCode=0 Oct 30 20:57:59.170621 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.170417 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" event={"ID":"fe019915-e577-45b7-96ae-ece4e2c1718b","Type":"ContainerDied","Data":"5fad1dfd2189542806e4042504ee1037bf888193bc76287248d182a304e95919"} Oct 30 20:57:59.172572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.172542 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" event={"ID":"d80ded81-5681-4a8c-a20a-d45a383d423b","Type":"ContainerDied","Data":"c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d"} Oct 30 20:57:59.172572 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.172562 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh" Oct 30 20:57:59.172803 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:57:59.172589 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2168d1e071457280d59c104bde646161b2446613f5ae8304778c769b5c6281d" Oct 30 20:58:00.387176 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.387141 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:58:00.423720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.421203 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\") pod \"fe019915-e577-45b7-96ae-ece4e2c1718b\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " Oct 30 20:58:00.423720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.421443 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\" (UniqueName: \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\") pod \"fe019915-e577-45b7-96ae-ece4e2c1718b\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " Oct 30 20:58:00.423720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.421486 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\") pod \"fe019915-e577-45b7-96ae-ece4e2c1718b\" (UID: \"fe019915-e577-45b7-96ae-ece4e2c1718b\") " Oct 30 20:58:00.423720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.422596 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca" pod "fe019915-e577-45b7-96ae-ece4e2c1718b" (UID: "fe019915-e577-45b7-96ae-ece4e2c1718b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:00.470772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.467832 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca" pod "fe019915-e577-45b7-96ae-ece4e2c1718b" (UID: "fe019915-e577-45b7-96ae-ece4e2c1718b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:00.470772 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.467889 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz" pod "fe019915-e577-45b7-96ae-ece4e2c1718b" (UID: "fe019915-e577-45b7-96ae-ece4e2c1718b"). InnerVolumeSpecName "kube-api-access-5brvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:00.552246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.552214 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5brvz\" (UniqueName: \"kubernetes.io/projected/fe019915-e577-45b7-96ae-ece4e2c1718b-kube-api-access-5brvz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.552246 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.552244 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.552504 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:00.552256 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe019915-e577-45b7-96ae-ece4e2c1718b-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.926150 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:00.926109 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:58:01.207630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:01.207603 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" Oct 30 20:58:01.207630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:01.207619 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl" event={"ID":"fe019915-e577-45b7-96ae-ece4e2c1718b","Type":"ContainerDied","Data":"f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145"} Oct 30 20:58:01.207874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:01.207697 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24badd0b7701462afeb9b50160dafd8174e3c41b3b233e5dae9cdf69392d145" Oct 30 20:58:02.447686 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.441821 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" containerID="cri-o://6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546" gracePeriod=45 Oct 30 20:58:02.451719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.449569 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": EOF" Oct 30 20:58:02.452300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.452009 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" containerID="cri-o://c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca" gracePeriod=45 Oct 30 20:58:02.453762 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.453695 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" containerID="cri-o://38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91" gracePeriod=45 Oct 30 20:58:02.767397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.767369 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:58:02.824630 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.824588 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:58:02.828680 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.828628 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:58:02.846793 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.846758 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.846980 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.846918 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.847178 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.847155 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.847288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.847221 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.847288 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.847279 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\" (UniqueName: \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.847858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.847833 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.848251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.848231 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.848745 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.848728 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\") pod \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\" (UID: \"0c739b88-a880-4edb-81ab-3deb0ff4ad8e\") " Oct 30 20:58:02.849251 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.848631 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.850095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.850050 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.850213 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.850198 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.856086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.856054 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.856287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.856083 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "kube-api-access-rlvwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:02.856287 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.856092 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs" pod "0c739b88-a880-4edb-81ab-3deb0ff4ad8e" (UID: "0c739b88-a880-4edb-81ab-3deb0ff4ad8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:02.952411 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952368 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.952587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952426 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.952587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952512 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.952587 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952553 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.952817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952614 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.952817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952711 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\" (UniqueName: \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.952817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952767 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.952817 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952810 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.953022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952896 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\" (UniqueName: \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.953022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.952974 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.953134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.953058 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\" (UniqueName: \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\") pod \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\" (UID: \"90e57a0c-3c51-400a-8bfc-1ab17f276b05\") " Oct 30 20:58:02.953134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.953109 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\" (UniqueName: \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\") pod \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\" (UID: \"06f2bef3-0671-4bec-bb3d-c5bacab55cde\") " Oct 30 20:58:02.953242 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.953138 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.954051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954013 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvwj\" (UniqueName: \"kubernetes.io/projected/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-kube-api-access-rlvwj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.954051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954042 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.954051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954053 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.954348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954062 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.954348 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954070 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c739b88-a880-4edb-81ab-3deb0ff4ad8e-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.954498 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954380 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.954579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.954542 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.955860 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.955825 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.957706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.956806 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.957706 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.957152 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "kube-api-access-jwplb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:02.957879 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.957856 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.958448 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.958413 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "kube-api-access-wcj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:02.959259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.959232 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.959708 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.959682 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.961016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.960993 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs" pod "06f2bef3-0671-4bec-bb3d-c5bacab55cde" (UID: "06f2bef3-0671-4bec-bb3d-c5bacab55cde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:02.961598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:02.961564 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs" pod "90e57a0c-3c51-400a-8bfc-1ab17f276b05" (UID: "90e57a0c-3c51-400a-8bfc-1ab17f276b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057327 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057420 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057437 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057450 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057466 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcj5j\" (UniqueName: \"kubernetes.io/projected/90e57a0c-3c51-400a-8bfc-1ab17f276b05-kube-api-access-wcj5j\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057483 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057496 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057507 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwplb\" (UniqueName: \"kubernetes.io/projected/06f2bef3-0671-4bec-bb3d-c5bacab55cde-kube-api-access-jwplb\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057522 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f2bef3-0671-4bec-bb3d-c5bacab55cde-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057536 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90e57a0c-3c51-400a-8bfc-1ab17f276b05-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.059682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.057549 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f2bef3-0671-4bec-bb3d-c5bacab55cde-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.230106 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.230072 5855 generic.go:356] "Generic (PLEG): container finished" podID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerID="6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546" exitCode=0 Oct 30 20:58:03.230309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.230156 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" Oct 30 20:58:03.230309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.230170 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" event={"ID":"0c739b88-a880-4edb-81ab-3deb0ff4ad8e","Type":"ContainerDied","Data":"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546"} Oct 30 20:58:03.230309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.230225 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc" event={"ID":"0c739b88-a880-4edb-81ab-3deb0ff4ad8e","Type":"ContainerDied","Data":"e924b7f60ae960bbd4f8aa3e822423a277542c8205735f39c03bda1b79a342fa"} Oct 30 20:58:03.230309 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.230247 5855 scope.go:122] "RemoveContainer" containerID="6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546" Oct 30 20:58:03.232610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.232582 5855 generic.go:356] "Generic (PLEG): container finished" podID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerID="38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91" exitCode=0 Oct 30 20:58:03.232750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.232627 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" Oct 30 20:58:03.232750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.232643 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" event={"ID":"90e57a0c-3c51-400a-8bfc-1ab17f276b05","Type":"ContainerDied","Data":"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91"} Oct 30 20:58:03.232750 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.232722 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm" event={"ID":"90e57a0c-3c51-400a-8bfc-1ab17f276b05","Type":"ContainerDied","Data":"8ef09d99765a65ccae88406e9353901ae8dff1c62ae9967821c3918efbcd1080"} Oct 30 20:58:03.234615 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.234591 5855 generic.go:356] "Generic (PLEG): container finished" podID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerID="c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca" exitCode=0 Oct 30 20:58:03.234784 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.234771 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" event={"ID":"06f2bef3-0671-4bec-bb3d-c5bacab55cde","Type":"ContainerDied","Data":"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca"} Oct 30 20:58:03.234858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.234799 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" event={"ID":"06f2bef3-0671-4bec-bb3d-c5bacab55cde","Type":"ContainerDied","Data":"14ae6509cdc72aad0ede5f4dd95f412f8b72c6af019681a84fe60982519bbd28"} Oct 30 20:58:03.234858 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.234847 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv" Oct 30 20:58:03.254479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.254457 5855 scope.go:122] "RemoveContainer" containerID="6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546" Oct 30 20:58:03.254936 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:03.254907 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546\": container with ID starting with 6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546 not found: ID does not exist" containerID="6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546" Oct 30 20:58:03.255041 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.254949 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546"} err="failed to get container status \"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546\": rpc error: code = NotFound desc = could not find container \"6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546\": container with ID starting with 6e212c34fd2e0ae66a42f7cfc4f5ae8cecfdc4387b7f345ee367eb90d5c07546 not found: ID does not exist" Oct 30 20:58:03.255041 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.254981 5855 scope.go:122] "RemoveContainer" containerID="38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91" Oct 30 20:58:03.266255 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.266237 5855 scope.go:122] "RemoveContainer" containerID="38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91" Oct 30 20:58:03.266576 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:03.266554 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91\": container with ID starting with 38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91 not found: ID does not exist" containerID="38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91" Oct 30 20:58:03.266628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.266586 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91"} err="failed to get container status \"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91\": rpc error: code = NotFound desc = could not find container \"38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91\": container with ID starting with 38f70eeade870645e11c1238dea44797c76a0aad49888aa99b23c58f49a87d91 not found: ID does not exist" Oct 30 20:58:03.266628 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.266606 5855 scope.go:122] "RemoveContainer" containerID="c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca" Oct 30 20:58:03.279231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.279210 5855 scope.go:122] "RemoveContainer" containerID="c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca" Oct 30 20:58:03.279569 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:03.279537 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca\": container with ID starting with c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca not found: ID does not exist" containerID="c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca" Oct 30 20:58:03.279721 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.279582 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca"} err="failed to get container status \"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca\": rpc error: code = NotFound desc = could not find container \"c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca\": container with ID starting with c5a4faacab8ea5ccbe3299bcfa497a9a9f89a74791ac0ecf8f6e4fcba09d2fca not found: ID does not exist" Oct 30 20:58:03.282501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.282473 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:58:03.290253 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.290225 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-2-w0-ad7ff0-kserve-588d747d9b-lfnmc"] Oct 30 20:58:03.297501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.297444 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:58:03.304461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.304433 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/simulator-w0-ad7ff0-kserve-5d78796f45-n2jzv"] Oct 30 20:58:03.322552 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.322450 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:58:03.327781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:03.327752 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/distinct-w0-ad7ff0-kserve-56cf7f77c9-82kcm"] Oct 30 20:58:04.780420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:04.780379 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" path="/var/lib/kubelet/pods/06f2bef3-0671-4bec-bb3d-c5bacab55cde/volumes" Oct 30 20:58:04.780924 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:04.780911 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" path="/var/lib/kubelet/pods/0c739b88-a880-4edb-81ab-3deb0ff4ad8e/volumes" Oct 30 20:58:04.781344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:04.781326 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" path="/var/lib/kubelet/pods/90e57a0c-3c51-400a-8bfc-1ab17f276b05/volumes" Oct 30 20:58:05.491101 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491068 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8"] Oct 30 20:58:05.491704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491678 5855 cpu_manager.go:414] "containerMap: removing container" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" Oct 30 20:58:05.491752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491706 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" Oct 30 20:58:05.491752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491734 5855 cpu_manager.go:414] "containerMap: removing container" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" Oct 30 20:58:05.491752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491740 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" Oct 30 20:58:05.491752 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491750 5855 cpu_manager.go:414] "containerMap: removing container" podUID="fe019915-e577-45b7-96ae-ece4e2c1718b" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491755 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe019915-e577-45b7-96ae-ece4e2c1718b" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491765 5855 cpu_manager.go:414] "containerMap: removing container" podUID="543c1b58-56e7-4c66-a866-4e132fc24c08" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491770 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543c1b58-56e7-4c66-a866-4e132fc24c08" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491776 5855 cpu_manager.go:414] "containerMap: removing container" podUID="d80ded81-5681-4a8c-a20a-d45a383d423b" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491781 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80ded81-5681-4a8c-a20a-d45a383d423b" containerName="revoke-keys" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491789 5855 cpu_manager.go:414] "containerMap: removing container" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491794 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" Oct 30 20:58:05.491888 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491882 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="543c1b58-56e7-4c66-a866-4e132fc24c08" containerName="revoke-keys" Oct 30 20:58:05.492134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491896 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80ded81-5681-4a8c-a20a-d45a383d423b" containerName="revoke-keys" Oct 30 20:58:05.492134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491903 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c739b88-a880-4edb-81ab-3deb0ff4ad8e" containerName="main" Oct 30 20:58:05.492134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491913 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e57a0c-3c51-400a-8bfc-1ab17f276b05" containerName="main" Oct 30 20:58:05.492134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491921 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f2bef3-0671-4bec-bb3d-c5bacab55cde" containerName="main" Oct 30 20:58:05.492134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.491927 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe019915-e577-45b7-96ae-ece4e2c1718b" containerName="revoke-keys" Oct 30 20:58:05.497749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.497723 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.504826 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.504798 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8"] Oct 30 20:58:05.505641 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.505617 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:58:05.505969 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.505746 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:58:05.506131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.505922 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:05.506131 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.505924 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:58:05.601396 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.601361 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tvk\" (UniqueName: \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.601563 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.601483 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.601766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.601749 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.703028 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.702988 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.703256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.703105 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tvk\" (UniqueName: \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.703256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.703165 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.703736 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.703711 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.706056 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.706028 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.716768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.716733 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tvk\" (UniqueName: \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\") pod \"maas-api-revoke-sub-7ace53c0-qnfm8\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:05.824051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:05.824017 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:06.047409 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.047184 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" containerID="cri-o://d0b663e335ad641649fa638c1d65e29881009f479d1001566d06bbcb6ba894bd" gracePeriod=45 Oct 30 20:58:06.064070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.063878 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" containerID="cri-o://b1ab7cca90762739734527567c0c5fa6413e8b22e517e790158df7118f004f99" gracePeriod=45 Oct 30 20:58:06.069289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.069081 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused" Oct 30 20:58:06.225550 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.225513 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8"] Oct 30 20:58:06.273705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.272607 5855 generic.go:356] "Generic (PLEG): container finished" podID="22db12c6-81be-4871-829b-111f9ec4619f" containerID="b1ab7cca90762739734527567c0c5fa6413e8b22e517e790158df7118f004f99" exitCode=0 Oct 30 20:58:06.273705 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.272940 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" event={"ID":"22db12c6-81be-4871-829b-111f9ec4619f","Type":"ContainerDied","Data":"b1ab7cca90762739734527567c0c5fa6413e8b22e517e790158df7118f004f99"} Oct 30 20:58:06.275802 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.275775 5855 generic.go:356] "Generic (PLEG): container finished" podID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerID="d0b663e335ad641649fa638c1d65e29881009f479d1001566d06bbcb6ba894bd" exitCode=0 Oct 30 20:58:06.276196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.276176 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" event={"ID":"093ec32c-630b-4b5b-a1a0-6cd42f66a396","Type":"ContainerDied","Data":"d0b663e335ad641649fa638c1d65e29881009f479d1001566d06bbcb6ba894bd"} Oct 30 20:58:06.277992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.277968 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" event={"ID":"525da2a4-998a-4f8b-a5d7-83d51dde8c31","Type":"ContainerStarted","Data":"66b3db91e7c93988e70ab60441012fa05378de65756d4d0af9bc1d14941716bd"} Oct 30 20:58:06.383954 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.383673 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:58:06.413693 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.413372 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:58:06.538047 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538013 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538115 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538135 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538163 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\" (UniqueName: \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538207 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538191 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538435 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538277 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\" (UniqueName: \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538435 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538341 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538435 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538395 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.538435 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538407 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538455 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\" (UniqueName: \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538483 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538511 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538523 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\" (UniqueName: \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\") pod \"22db12c6-81be-4871-829b-111f9ec4619f\" (UID: \"22db12c6-81be-4871-829b-111f9ec4619f\") " Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538566 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538623 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\") pod \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\" (UID: \"093ec32c-630b-4b5b-a1a0-6cd42f66a396\") " Oct 30 20:58:06.538995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538699 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.538995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538726 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.538995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.538967 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.539766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539736 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.539895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539770 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.539895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539790 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.539895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539804 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.539895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539817 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.539895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.539831 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.541375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541348 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.541375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541370 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "kube-api-access-x5v4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:06.541543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541400 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:06.541543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541414 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.541543 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541438 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799" pod "093ec32c-630b-4b5b-a1a0-6cd42f66a396" (UID: "093ec32c-630b-4b5b-a1a0-6cd42f66a396"). InnerVolumeSpecName "kube-api-access-fb799". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:06.541830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.541806 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs" pod "22db12c6-81be-4871-829b-111f9ec4619f" (UID: "22db12c6-81be-4871-829b-111f9ec4619f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:06.644026 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.643927 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093ec32c-630b-4b5b-a1a0-6cd42f66a396-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.644259 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.644242 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22db12c6-81be-4871-829b-111f9ec4619f-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.644353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.644343 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093ec32c-630b-4b5b-a1a0-6cd42f66a396-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.644436 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.644427 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22db12c6-81be-4871-829b-111f9ec4619f-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.644518 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.644507 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb799\" (UniqueName: \"kubernetes.io/projected/093ec32c-630b-4b5b-a1a0-6cd42f66a396-kube-api-access-fb799\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.644603 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:06.644591 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5v4m\" (UniqueName: \"kubernetes.io/projected/22db12c6-81be-4871-829b-111f9ec4619f-kube-api-access-x5v4m\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:07.357042 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.356927 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:07.373756 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.373716 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" event={"ID":"525da2a4-998a-4f8b-a5d7-83d51dde8c31","Type":"ContainerStarted","Data":"0ed28654bb19ef4cf62b228a980544ff0cedd96c15304c0adb9cf1da70aa8c12"} Oct 30 20:58:07.432007 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.428141 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" event={"ID":"22db12c6-81be-4871-829b-111f9ec4619f","Type":"ContainerDied","Data":"5a1e6cacbbf6bdbb9f2a51ae2f0565c354618c4786160576fffe6e55b00a8e65"} Oct 30 20:58:07.432007 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.428209 5855 scope.go:122] "RemoveContainer" containerID="b1ab7cca90762739734527567c0c5fa6413e8b22e517e790158df7118f004f99" Oct 30 20:58:07.432007 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.428358 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k" Oct 30 20:58:07.434369 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.434337 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" event={"ID":"093ec32c-630b-4b5b-a1a0-6cd42f66a396","Type":"ContainerDied","Data":"4a73e54561c297ce40782d2928a9c31f4871fbde3f38d1fb78ad3c1cace51ba7"} Oct 30 20:58:07.434617 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.434600 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7" Oct 30 20:58:07.468699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.468641 5855 scope.go:122] "RemoveContainer" containerID="d0b663e335ad641649fa638c1d65e29881009f479d1001566d06bbcb6ba894bd" Oct 30 20:58:07.478917 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.478862 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" podStartSLOduration=2.47884359 podStartE2EDuration="2.47884359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:07.425303718 +0000 UTC m=+1100.784016466" watchObservedRunningTime="2026-10-30 20:58:07.47884359 +0000 UTC m=+1100.837556339" Oct 30 20:58:07.486546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.486523 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:58:07.492812 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.492780 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-2-w3-cb60a2-kserve-659cb75957-2z9v7"] Oct 30 20:58:07.501112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.501085 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:58:07.507492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:07.507464 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/premium-w3-cb60a2-kserve-74d58ccdc-g958k"] Oct 30 20:58:08.446782 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:08.446746 5855 generic.go:356] "Generic (PLEG): container finished" podID="525da2a4-998a-4f8b-a5d7-83d51dde8c31" containerID="0ed28654bb19ef4cf62b228a980544ff0cedd96c15304c0adb9cf1da70aa8c12" exitCode=0 Oct 30 20:58:08.447311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:08.446839 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" event={"ID":"525da2a4-998a-4f8b-a5d7-83d51dde8c31","Type":"ContainerDied","Data":"0ed28654bb19ef4cf62b228a980544ff0cedd96c15304c0adb9cf1da70aa8c12"} Oct 30 20:58:08.782103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:08.782013 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" path="/var/lib/kubelet/pods/093ec32c-630b-4b5b-a1a0-6cd42f66a396/volumes" Oct 30 20:58:08.782556 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:08.782538 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22db12c6-81be-4871-829b-111f9ec4619f" path="/var/lib/kubelet/pods/22db12c6-81be-4871-829b-111f9ec4619f/volumes" Oct 30 20:58:09.632157 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.632127 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:09.726460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.726375 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\" (UniqueName: \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\") pod \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " Oct 30 20:58:09.726460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.726418 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\") pod \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " Oct 30 20:58:09.726691 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.726497 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\") pod \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\" (UID: \"525da2a4-998a-4f8b-a5d7-83d51dde8c31\") " Oct 30 20:58:09.726853 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.726829 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca" pod "525da2a4-998a-4f8b-a5d7-83d51dde8c31" (UID: "525da2a4-998a-4f8b-a5d7-83d51dde8c31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:09.726957 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.726944 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:09.728840 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.728812 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca" pod "525da2a4-998a-4f8b-a5d7-83d51dde8c31" (UID: "525da2a4-998a-4f8b-a5d7-83d51dde8c31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:09.728971 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.728875 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk" pod "525da2a4-998a-4f8b-a5d7-83d51dde8c31" (UID: "525da2a4-998a-4f8b-a5d7-83d51dde8c31"). InnerVolumeSpecName "kube-api-access-78tvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:09.828850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.828810 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tvk\" (UniqueName: \"kubernetes.io/projected/525da2a4-998a-4f8b-a5d7-83d51dde8c31-kube-api-access-78tvk\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:09.828850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:09.828843 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525da2a4-998a-4f8b-a5d7-83d51dde8c31-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:10.468862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:10.468828 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" Oct 30 20:58:10.468862 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:10.468837 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8" event={"ID":"525da2a4-998a-4f8b-a5d7-83d51dde8c31","Type":"ContainerDied","Data":"66b3db91e7c93988e70ab60441012fa05378de65756d4d0af9bc1d14941716bd"} Oct 30 20:58:10.469048 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:10.468886 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b3db91e7c93988e70ab60441012fa05378de65756d4d0af9bc1d14941716bd" Oct 30 20:58:12.444683 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:12.437721 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg"] Oct 30 20:58:12.479681 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:12.477882 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ad7ff0-z44tg"] Oct 30 20:58:12.842694 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:12.841785 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83a039c-0218-43be-902e-6f5aeb063647" path="/var/lib/kubelet/pods/d83a039c-0218-43be-902e-6f5aeb063647/volumes" Oct 30 20:58:15.290027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:15.289994 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7"] Oct 30 20:58:15.305547 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:15.305509 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-cb60a2-mcwg7"] Oct 30 20:58:16.779854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:16.779815 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20120e07-b1c1-4dc9-a3cc-03be73dd73dc" path="/var/lib/kubelet/pods/20120e07-b1c1-4dc9-a3cc-03be73dd73dc/volumes" Oct 30 20:58:20.481743 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.481701 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:58:20.483889 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.483847 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" containerID="cri-o://1cc6e9125e0593939b555a7493364436c7bee76443098ee60bca60c05c1382f4" gracePeriod=30 Oct 30 20:58:20.485200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.484751 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:58:20.485200 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.484937 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" containerID="cri-o://e1be3b466fa587b1d64c76ab7bb201f68fe980f3b74ca3887663e93b9be73c00" gracePeriod=30 Oct 30 20:58:20.494815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.493262 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:58:20.494815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.493475 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" containerID="cri-o://2a6248e0f32b171133e72e80f259cc26cc3427bc760cffbd7152a69aeda489b9" gracePeriod=30 Oct 30 20:58:20.499668 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.499584 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:58:20.500326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:20.500268 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" containerID="cri-o://e05ea7a419f7ecc2481e96f7ddb4d07e2d47c25e995ee88db511115740685f49" gracePeriod=30 Oct 30 20:58:24.500676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.498333 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:58:24.500676 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.498674 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" containerID="cri-o://47eb4530b2ffc818c5c276de81c0e7ea3af0733b01328f3a67ea9a31cfec1a66" gracePeriod=30 Oct 30 20:58:24.518761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.518722 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:58:24.518969 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.518804 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:58:24.520804 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.519016 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" containerID="cri-o://e98a81def1e70644b7092547b360db36e70969672302d2b2ca894f21a819cd44" gracePeriod=30 Oct 30 20:58:24.520804 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.519793 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" containerID="cri-o://00dcdf6a5ba33848ce2a6cb86953b2aac01afe2494b0a4794fd0468bf5cb9eb5" gracePeriod=30 Oct 30 20:58:24.530685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.527430 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:58:24.530685 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.528124 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" containerID="cri-o://7fed4ef894fdbcad8bfff29c119a73c2ec9580320fb11db41129a6303c4eeb6a" gracePeriod=30 Oct 30 20:58:24.879117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.878805 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:24.879117 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:24.878863 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:25.232712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.232587 5855 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:25.232712 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.232679 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:25.982912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.980926 5855 generic.go:356] "Generic (PLEG): container finished" podID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerID="e1be3b466fa587b1d64c76ab7bb201f68fe980f3b74ca3887663e93b9be73c00" exitCode=0 Oct 30 20:58:25.982912 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.981058 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" event={"ID":"72a54c98-8ed7-4691-b3f2-c6000eab4f87","Type":"ContainerDied","Data":"e1be3b466fa587b1d64c76ab7bb201f68fe980f3b74ca3887663e93b9be73c00"} Oct 30 20:58:25.983774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.982978 5855 generic.go:356] "Generic (PLEG): container finished" podID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerID="1cc6e9125e0593939b555a7493364436c7bee76443098ee60bca60c05c1382f4" exitCode=0 Oct 30 20:58:25.983774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.983130 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" event={"ID":"2fbc8764-d0d6-498e-a3a9-6436433a249c","Type":"ContainerDied","Data":"1cc6e9125e0593939b555a7493364436c7bee76443098ee60bca60c05c1382f4"} Oct 30 20:58:25.984839 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.984818 5855 generic.go:356] "Generic (PLEG): container finished" podID="55fb2495-6764-4bea-8c5e-905417748912" containerID="2a6248e0f32b171133e72e80f259cc26cc3427bc760cffbd7152a69aeda489b9" exitCode=0 Oct 30 20:58:25.985419 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.985026 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" event={"ID":"55fb2495-6764-4bea-8c5e-905417748912","Type":"ContainerDied","Data":"2a6248e0f32b171133e72e80f259cc26cc3427bc760cffbd7152a69aeda489b9"} Oct 30 20:58:25.997924 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.997875 5855 generic.go:356] "Generic (PLEG): container finished" podID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerID="e05ea7a419f7ecc2481e96f7ddb4d07e2d47c25e995ee88db511115740685f49" exitCode=0 Oct 30 20:58:25.998345 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:25.998324 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" event={"ID":"be5d6f5e-154c-44ac-b6ac-4547a612f485","Type":"ContainerDied","Data":"e05ea7a419f7ecc2481e96f7ddb4d07e2d47c25e995ee88db511115740685f49"} Oct 30 20:58:26.036129 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.036101 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:58:26.155313 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155271 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155337 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155437 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155475 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155517 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155548 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155625 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155866 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155844 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.155936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.155923 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\") pod \"2fbc8764-d0d6-498e-a3a9-6436433a249c\" (UID: \"2fbc8764-d0d6-498e-a3a9-6436433a249c\") " Oct 30 20:58:26.158319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.158286 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.159530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.159054 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:58:26.159530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.159352 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.160354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.159770 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:26.160354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.159941 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.160354 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.160007 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.165876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.165815 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.166063 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.166006 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:26.167903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.167227 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.170785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.170015 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz" pod "2fbc8764-d0d6-498e-a3a9-6436433a249c" (UID: "2fbc8764-d0d6-498e-a3a9-6436433a249c"). InnerVolumeSpecName "kube-api-access-4s5lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.173091 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.173067 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:58:26.181404 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.181290 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:58:26.259952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.259859 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.259952 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.259944 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260016 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260092 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260136 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260126 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260166 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.260236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260219 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260295 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260265 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260305 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.260414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260386 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260444 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.260501 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260473 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.260600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260509 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.260600 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260545 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260619 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260682 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260731 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.260916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260805 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260835 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260869 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.260916 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260906 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.261151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.260975 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.261151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261030 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.261151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261054 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\") pod \"be5d6f5e-154c-44ac-b6ac-4547a612f485\" (UID: \"be5d6f5e-154c-44ac-b6ac-4547a612f485\") " Oct 30 20:58:26.261151 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261094 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\") pod \"55fb2495-6764-4bea-8c5e-905417748912\" (UID: \"55fb2495-6764-4bea-8c5e-905417748912\") " Oct 30 20:58:26.261366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261174 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.261366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261224 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\") pod \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\" (UID: \"72a54c98-8ed7-4691-b3f2-c6000eab4f87\") " Oct 30 20:58:26.262016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.261990 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.266845 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.266821 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.267466 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267449 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.267610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267598 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.267749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267733 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5lz\" (UniqueName: \"kubernetes.io/projected/2fbc8764-d0d6-498e-a3a9-6436433a249c-kube-api-access-4s5lz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.267838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267827 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.267922 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267912 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fbc8764-d0d6-498e-a3a9-6436433a249c-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.268020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268009 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.268103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268091 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fbc8764-d0d6-498e-a3a9-6436433a249c-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.268187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.264418 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.268275 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267147 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:26.268347 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267187 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:26.268486 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268375 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:26.268692 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267404 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.271184 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.271162 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.271377 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.271348 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.271613 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.271592 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.272468 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.267959 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268106 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268805 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268829 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.268734 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.272826 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "kube-api-access-7jwvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.272891 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.272851 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.273307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.272894 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:26.273307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.273128 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "kube-api-access-qntdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.273307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.273194 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.273574 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.273541 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.274471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.274445 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:26.274574 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.274526 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.278112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.278073 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:26.278112 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.278069 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:26.278342 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.278321 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj" pod "be5d6f5e-154c-44ac-b6ac-4547a612f485" (UID: "be5d6f5e-154c-44ac-b6ac-4547a612f485"). InnerVolumeSpecName "kube-api-access-wqdpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.278828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.278803 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token" pod "55fb2495-6764-4bea-8c5e-905417748912" (UID: "55fb2495-6764-4bea-8c5e-905417748912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.279016 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.278986 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token" pod "72a54c98-8ed7-4691-b3f2-c6000eab4f87" (UID: "72a54c98-8ed7-4691-b3f2-c6000eab4f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:26.369282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369248 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369278 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369282 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369287 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369296 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qntdd\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-kube-api-access-qntdd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369307 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqdpj\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-kube-api-access-wqdpj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369315 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369323 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369333 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369342 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369350 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369358 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55fb2495-6764-4bea-8c5e-905417748912-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369365 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369373 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369381 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369389 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369396 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369404 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5d6f5e-154c-44ac-b6ac-4547a612f485-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369412 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369421 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369429 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369437 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55fb2495-6764-4bea-8c5e-905417748912-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369445 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5d6f5e-154c-44ac-b6ac-4547a612f485-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369454 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwvp\" (UniqueName: \"kubernetes.io/projected/72a54c98-8ed7-4691-b3f2-c6000eab4f87-kube-api-access-7jwvp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369462 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55fb2495-6764-4bea-8c5e-905417748912-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369470 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5d6f5e-154c-44ac-b6ac-4547a612f485-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369478 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55fb2495-6764-4bea-8c5e-905417748912-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.369519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:26.369486 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a54c98-8ed7-4691-b3f2-c6000eab4f87-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:27.011579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.011543 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" Oct 30 20:58:27.011579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.011566 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp" event={"ID":"72a54c98-8ed7-4691-b3f2-c6000eab4f87","Type":"ContainerDied","Data":"b243369bcb84af1b9360dbae2bf7ba0c5cfc40d28eb095d65365d2f0b9ce6a97"} Oct 30 20:58:27.012169 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.011692 5855 scope.go:122] "RemoveContainer" containerID="e1be3b466fa587b1d64c76ab7bb201f68fe980f3b74ca3887663e93b9be73c00" Oct 30 20:58:27.016220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.014947 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" Oct 30 20:58:27.016220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.015012 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf" event={"ID":"2fbc8764-d0d6-498e-a3a9-6436433a249c","Type":"ContainerDied","Data":"3a523f153a96e90748d5037c080868a70c2f79505370c714c64761614d91f8df"} Oct 30 20:58:27.018605 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.018572 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" event={"ID":"55fb2495-6764-4bea-8c5e-905417748912","Type":"ContainerDied","Data":"16a2844141cf67e7f9f4df553061ba63c3a6310f686e6160ff3170934ab4d4bf"} Oct 30 20:58:27.018753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.018695 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc" Oct 30 20:58:27.021260 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.021237 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" event={"ID":"be5d6f5e-154c-44ac-b6ac-4547a612f485","Type":"ContainerDied","Data":"0d80a3d6fa1a15f3f46b833d85a5a45daeaf2d46b4a1a6ea770fcd3452e80f7a"} Oct 30 20:58:27.021372 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.021339 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn" Oct 30 20:58:27.031391 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.031370 5855 scope.go:122] "RemoveContainer" containerID="1cc6e9125e0593939b555a7493364436c7bee76443098ee60bca60c05c1382f4" Oct 30 20:58:27.040783 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.040757 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:58:27.045070 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.045049 5855 scope.go:122] "RemoveContainer" containerID="2a6248e0f32b171133e72e80f259cc26cc3427bc760cffbd7152a69aeda489b9" Oct 30 20:58:27.045326 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.045305 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-jz4vp"] Oct 30 20:58:27.056666 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.056628 5855 scope.go:122] "RemoveContainer" containerID="e05ea7a419f7ecc2481e96f7ddb4d07e2d47c25e995ee88db511115740685f49" Oct 30 20:58:27.057575 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.057557 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:58:27.062481 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.062460 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-b2jgc"] Oct 30 20:58:27.077193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.077167 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:58:27.118928 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.118848 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-9cfxn"] Oct 30 20:58:27.143675 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.140002 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:58:27.163774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.163549 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kcwjf"] Oct 30 20:58:27.302511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.302426 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:27.302511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.302488 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:27.463749 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:27.463700 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:58:28.788038 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.788002 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" path="/var/lib/kubelet/pods/2fbc8764-d0d6-498e-a3a9-6436433a249c/volumes" Oct 30 20:58:28.789391 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.789364 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fb2495-6764-4bea-8c5e-905417748912" path="/var/lib/kubelet/pods/55fb2495-6764-4bea-8c5e-905417748912/volumes" Oct 30 20:58:28.790240 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.790218 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" path="/var/lib/kubelet/pods/72a54c98-8ed7-4691-b3f2-c6000eab4f87/volumes" Oct 30 20:58:28.791434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.791415 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" path="/var/lib/kubelet/pods/be5d6f5e-154c-44ac-b6ac-4547a612f485/volumes" Oct 30 20:58:28.803865 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.803818 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:28.804197 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.804151 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:28.982642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.982180 5855 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:28.982642 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:28.982265 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:30.194376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.194134 5855 generic.go:356] "Generic (PLEG): container finished" podID="fe64f471-65f4-49d9-b086-5d98363146a4" containerID="7fed4ef894fdbcad8bfff29c119a73c2ec9580320fb11db41129a6303c4eeb6a" exitCode=0 Oct 30 20:58:30.194376 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.194261 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" event={"ID":"fe64f471-65f4-49d9-b086-5d98363146a4","Type":"ContainerDied","Data":"7fed4ef894fdbcad8bfff29c119a73c2ec9580320fb11db41129a6303c4eeb6a"} Oct 30 20:58:30.197324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.197122 5855 generic.go:356] "Generic (PLEG): container finished" podID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerID="e98a81def1e70644b7092547b360db36e70969672302d2b2ca894f21a819cd44" exitCode=0 Oct 30 20:58:30.197324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.197257 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" event={"ID":"cec75fe6-1787-4b66-96a5-7c2412ae7414","Type":"ContainerDied","Data":"e98a81def1e70644b7092547b360db36e70969672302d2b2ca894f21a819cd44"} Oct 30 20:58:30.200353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.200121 5855 generic.go:356] "Generic (PLEG): container finished" podID="9917567c-d501-4d54-9067-a259a98d9b44" containerID="00dcdf6a5ba33848ce2a6cb86953b2aac01afe2494b0a4794fd0468bf5cb9eb5" exitCode=0 Oct 30 20:58:30.200353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.200203 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerDied","Data":"00dcdf6a5ba33848ce2a6cb86953b2aac01afe2494b0a4794fd0468bf5cb9eb5"} Oct 30 20:58:30.200353 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.200253 5855 scope.go:122] "RemoveContainer" containerID="d610a9b5a93f4e5bc94d2dee0ed0537fe8a2151a6e3d624275237d999431cd3b" Oct 30 20:58:30.217463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.215773 5855 generic.go:356] "Generic (PLEG): container finished" podID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerID="47eb4530b2ffc818c5c276de81c0e7ea3af0733b01328f3a67ea9a31cfec1a66" exitCode=0 Oct 30 20:58:30.217463 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.215949 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" event={"ID":"37a2f120-39e0-43aa-8490-e4006c7f636c","Type":"ContainerDied","Data":"47eb4530b2ffc818c5c276de81c0e7ea3af0733b01328f3a67ea9a31cfec1a66"} Oct 30 20:58:30.697807 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.684609 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.774752 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.774830 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.774953 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.775059 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.778151 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.794281 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.778228 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.802084 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.802149 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.802196 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.802279 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.802318 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\") pod \"cec75fe6-1787-4b66-96a5-7c2412ae7414\" (UID: \"cec75fe6-1787-4b66-96a5-7c2412ae7414\") " Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.803090 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.803116 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.803790 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.805279 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.804634 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.818742 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.818275 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:58:30.824780 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.824711 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "kube-api-access-zgqvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:30.834699 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.827157 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:30.899220 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.899164 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910015 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910121 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910159 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910241 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910315 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910369 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910416 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910492 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.910529 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\") pod \"fe64f471-65f4-49d9-b086-5d98363146a4\" (UID: \"fe64f471-65f4-49d9-b086-5d98363146a4\") " Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.911227 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqvp\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-kube-api-access-zgqvp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.911256 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.911274 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.911290 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec75fe6-1787-4b66-96a5-7c2412ae7414-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.912036 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.911304 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec75fe6-1787-4b66-96a5-7c2412ae7414-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:30.925991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.913625 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.925991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.914267 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.925991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.914505 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:30.925991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.914818 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:30.925991 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:30.915278 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.009562 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.009450 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014561 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014598 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014615 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014631 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe64f471-65f4-49d9-b086-5d98363146a4-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014646 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.014710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.014676 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.043594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.026898 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.043594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.026993 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.043594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.027222 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token" pod "cec75fe6-1787-4b66-96a5-7c2412ae7414" (UID: "cec75fe6-1787-4b66-96a5-7c2412ae7414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.043594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.043346 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.043594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.043540 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9" pod "fe64f471-65f4-49d9-b086-5d98363146a4" (UID: "fe64f471-65f4-49d9-b086-5d98363146a4"). InnerVolumeSpecName "kube-api-access-k7vj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.049622 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.048289 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:58:31.069958 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:31.069861 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:58:31.117598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.116811 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7vj9\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-kube-api-access-k7vj9\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.117598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.116848 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec75fe6-1787-4b66-96a5-7c2412ae7414-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.117598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.116866 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe64f471-65f4-49d9-b086-5d98363146a4-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.117598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.116880 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe64f471-65f4-49d9-b086-5d98363146a4-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.117598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.116894 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe64f471-65f4-49d9-b086-5d98363146a4-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.182561 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.182527 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:58:31.217672 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.217591 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.217728 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.217859 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218053 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218115 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218182 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.218216 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218219 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218256 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218294 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218325 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218366 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218399 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218446 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.218597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218571 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\") pod \"37a2f120-39e0-43aa-8490-e4006c7f636c\" (UID: \"37a2f120-39e0-43aa-8490-e4006c7f636c\") " Oct 30 20:58:31.219020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218632 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.219020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.218683 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.219818 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.219790 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.220018 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.219993 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.220193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.220169 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.220421 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.220398 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.220854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.220827 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.221028 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.220964 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.221719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.221448 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.222314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.222041 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.222420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.222402 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.222591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.222544 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.224567 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.224449 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.224825 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.224788 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.225597 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.225561 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.228716 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.227303 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "kube-api-access-qt8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.235601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.235561 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.237578 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.237542 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn" pod "37a2f120-39e0-43aa-8490-e4006c7f636c" (UID: "37a2f120-39e0-43aa-8490-e4006c7f636c"). InnerVolumeSpecName "kube-api-access-lwbdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.244093 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.243978 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" Oct 30 20:58:31.244344 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.244320 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx" event={"ID":"fe64f471-65f4-49d9-b086-5d98363146a4","Type":"ContainerDied","Data":"2f7c8ef83c3dc21cebe4f24cc2637822e8cd794e7e2636b74a467ef779f4e92d"} Oct 30 20:58:31.244510 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.244496 5855 scope.go:122] "RemoveContainer" containerID="7fed4ef894fdbcad8bfff29c119a73c2ec9580320fb11db41129a6303c4eeb6a" Oct 30 20:58:31.249744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.246600 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" event={"ID":"cec75fe6-1787-4b66-96a5-7c2412ae7414","Type":"ContainerDied","Data":"9b5d1db2b714f556c16d851697657c32a720c97fbdb7de6eda4bb94b007f7277"} Oct 30 20:58:31.249744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.246719 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp" Oct 30 20:58:31.256751 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.256721 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" event={"ID":"9917567c-d501-4d54-9067-a259a98d9b44","Type":"ContainerDied","Data":"e47cb3db973c112b306a9c7af41865ff1edde0fa5d049a26e2d7ac43589fa6f4"} Oct 30 20:58:31.256899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.256839 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh" Oct 30 20:58:31.264062 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.264032 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" event={"ID":"37a2f120-39e0-43aa-8490-e4006c7f636c","Type":"ContainerDied","Data":"08aa32f5ed8677bbf38c402d200bfd3f1a003ff9d0111489a24e97d60f7b8128"} Oct 30 20:58:31.264165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.264134 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj" Oct 30 20:58:31.266595 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.266503 5855 scope.go:122] "RemoveContainer" containerID="e98a81def1e70644b7092547b360db36e70969672302d2b2ca894f21a819cd44" Oct 30 20:58:31.309881 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.308121 5855 scope.go:122] "RemoveContainer" containerID="00dcdf6a5ba33848ce2a6cb86953b2aac01afe2494b0a4794fd0468bf5cb9eb5" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.320579 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.320637 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\") pod \"9917567c-d501-4d54-9067-a259a98d9b44\" (UID: \"9917567c-d501-4d54-9067-a259a98d9b44\") " Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321326 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321352 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8q2\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-kube-api-access-qt8q2\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321368 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321381 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9917567c-d501-4d54-9067-a259a98d9b44-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321395 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321408 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321421 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321434 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37a2f120-39e0-43aa-8490-e4006c7f636c-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321447 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321460 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321474 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321487 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321500 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwbdn\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-kube-api-access-lwbdn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321515 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37a2f120-39e0-43aa-8490-e4006c7f636c-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321528 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37a2f120-39e0-43aa-8490-e4006c7f636c-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.321592 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321540 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9917567c-d501-4d54-9067-a259a98d9b44-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.329855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.321911 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.329855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.326175 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:58:31.329855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.329024 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token" pod "9917567c-d501-4d54-9067-a259a98d9b44" (UID: "9917567c-d501-4d54-9067-a259a98d9b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.345089 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.345036 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-lxtsp"] Oct 30 20:58:31.345631 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.345609 5855 scope.go:122] "RemoveContainer" containerID="47eb4530b2ffc818c5c276de81c0e7ea3af0733b01328f3a67ea9a31cfec1a66" Oct 30 20:58:31.368753 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.368709 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:58:31.372596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.372567 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-t2kmx"] Oct 30 20:58:31.388746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.388669 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:58:31.395790 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.395753 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-cqbjj"] Oct 30 20:58:31.423063 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.423027 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9917567c-d501-4d54-9067-a259a98d9b44-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.423063 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.423057 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9917567c-d501-4d54-9067-a259a98d9b44-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.587025 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.586991 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:58:31.593192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:31.593151 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-mbxlh"] Oct 30 20:58:32.783397 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:32.783361 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" path="/var/lib/kubelet/pods/37a2f120-39e0-43aa-8490-e4006c7f636c/volumes" Oct 30 20:58:32.784085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:32.784063 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9917567c-d501-4d54-9067-a259a98d9b44" path="/var/lib/kubelet/pods/9917567c-d501-4d54-9067-a259a98d9b44/volumes" Oct 30 20:58:32.784719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:32.784693 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" path="/var/lib/kubelet/pods/cec75fe6-1787-4b66-96a5-7c2412ae7414/volumes" Oct 30 20:58:32.785834 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:32.785810 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" path="/var/lib/kubelet/pods/fe64f471-65f4-49d9-b086-5d98363146a4/volumes" Oct 30 20:58:33.028086 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.028054 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z"] Oct 30 20:58:33.031650 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.031618 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d11e2b1-ttx8z"] Oct 30 20:58:33.821967 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.821927 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8"] Oct 30 20:58:33.822434 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822421 5855 cpu_manager.go:414] "containerMap: removing container" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822435 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822454 5855 cpu_manager.go:414] "containerMap: removing container" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822459 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822468 5855 cpu_manager.go:414] "containerMap: removing container" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822472 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822478 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822480 5855 cpu_manager.go:414] "containerMap: removing container" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822485 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822495 5855 cpu_manager.go:414] "containerMap: removing container" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822499 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822507 5855 cpu_manager.go:414] "containerMap: removing container" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822511 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822522 5855 cpu_manager.go:414] "containerMap: removing container" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822526 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822534 5855 cpu_manager.go:414] "containerMap: removing container" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822538 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822551 5855 cpu_manager.go:414] "containerMap: removing container" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822555 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822560 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822564 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822578 5855 cpu_manager.go:414] "containerMap: removing container" podUID="525da2a4-998a-4f8b-a5d7-83d51dde8c31" containerName="revoke-keys" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822582 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525da2a4-998a-4f8b-a5d7-83d51dde8c31" containerName="revoke-keys" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822590 5855 cpu_manager.go:414] "containerMap: removing container" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822594 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822675 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="525da2a4-998a-4f8b-a5d7-83d51dde8c31" containerName="revoke-keys" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822684 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe64f471-65f4-49d9-b086-5d98363146a4" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822694 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="093ec32c-630b-4b5b-a1a0-6cd42f66a396" containerName="main" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822703 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822710 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="9917567c-d501-4d54-9067-a259a98d9b44" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822715 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec75fe6-1787-4b66-96a5-7c2412ae7414" containerName="istio-proxy" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822719 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="22db12c6-81be-4871-829b-111f9ec4619f" containerName="main" Oct 30 20:58:33.822707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822724 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a2f120-39e0-43aa-8490-e4006c7f636c" containerName="istio-proxy" Oct 30 20:58:33.823414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822729 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a54c98-8ed7-4691-b3f2-c6000eab4f87" containerName="istio-proxy" Oct 30 20:58:33.823414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822735 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fb2495-6764-4bea-8c5e-905417748912" containerName="istio-proxy" Oct 30 20:58:33.823414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822742 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5d6f5e-154c-44ac-b6ac-4547a612f485" containerName="istio-proxy" Oct 30 20:58:33.823414 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.822747 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbc8764-d0d6-498e-a3a9-6436433a249c" containerName="istio-proxy" Oct 30 20:58:33.828311 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.828282 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.833719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.833668 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8"] Oct 30 20:58:33.834189 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.834170 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:58:33.835810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.835772 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:33.835940 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.835784 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:58:33.836021 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.835806 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:58:33.853095 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.853054 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.853267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.853178 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.853267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.853215 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh8lt\" (UniqueName: \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.967269 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.967228 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.967737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.967545 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.967737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.967598 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh8lt\" (UniqueName: \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:33.990596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:33.990554 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:34.004314 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:34.004279 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:34.009856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:34.009636 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh8lt\" (UniqueName: \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\") pod \"maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:34.146943 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:34.146839 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:34.445398 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:58:34.445369 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26ee112_e6d6_454e_a165_7f60ae007df9.slice/crio-5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9 WatchSource:0}: Error finding container 5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9: Status 404 returned error can't find the container with id 5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9 Oct 30 20:58:34.445530 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:34.445505 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8"] Oct 30 20:58:34.783776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:34.783700 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252d1bc4-5112-442d-8299-73db4dff62c2" path="/var/lib/kubelet/pods/252d1bc4-5112-442d-8299-73db4dff62c2/volumes" Oct 30 20:58:35.323983 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:35.323947 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" event={"ID":"b26ee112-e6d6-454e-a165-7f60ae007df9","Type":"ContainerStarted","Data":"34b8ad6c1ad981c71a38b162bb44e5067f7ab69077620a6a75e9fcb70d869803"} Oct 30 20:58:35.324375 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:35.324007 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" event={"ID":"b26ee112-e6d6-454e-a165-7f60ae007df9","Type":"ContainerStarted","Data":"5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9"} Oct 30 20:58:35.345647 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:35.345597 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" podStartSLOduration=2.345585226 podStartE2EDuration="2.345585226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:35.343809196 +0000 UTC m=+1128.702521944" watchObservedRunningTime="2026-10-30 20:58:35.345585226 +0000 UTC m=+1128.704298000" Oct 30 20:58:36.335037 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:36.335005 5855 generic.go:356] "Generic (PLEG): container finished" podID="b26ee112-e6d6-454e-a165-7f60ae007df9" containerID="34b8ad6c1ad981c71a38b162bb44e5067f7ab69077620a6a75e9fcb70d869803" exitCode=0 Oct 30 20:58:36.335452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:36.335083 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" event={"ID":"b26ee112-e6d6-454e-a165-7f60ae007df9","Type":"ContainerDied","Data":"34b8ad6c1ad981c71a38b162bb44e5067f7ab69077620a6a75e9fcb70d869803"} Oct 30 20:58:37.572330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.572303 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:37.610166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.609404 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\") pod \"b26ee112-e6d6-454e-a165-7f60ae007df9\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " Oct 30 20:58:37.610166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.609515 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\" (UniqueName: \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\") pod \"b26ee112-e6d6-454e-a165-7f60ae007df9\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " Oct 30 20:58:37.610166 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.609695 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\") pod \"b26ee112-e6d6-454e-a165-7f60ae007df9\" (UID: \"b26ee112-e6d6-454e-a165-7f60ae007df9\") " Oct 30 20:58:37.610451 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.610371 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca" pod "b26ee112-e6d6-454e-a165-7f60ae007df9" (UID: "b26ee112-e6d6-454e-a165-7f60ae007df9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:37.623461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.623322 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca" pod "b26ee112-e6d6-454e-a165-7f60ae007df9" (UID: "b26ee112-e6d6-454e-a165-7f60ae007df9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:37.626020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.625915 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt" pod "b26ee112-e6d6-454e-a165-7f60ae007df9" (UID: "b26ee112-e6d6-454e-a165-7f60ae007df9"). InnerVolumeSpecName "kube-api-access-xh8lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:37.711133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.711091 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:37.711133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.711135 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26ee112-e6d6-454e-a165-7f60ae007df9-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:37.711404 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:37.711148 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh8lt\" (UniqueName: \"kubernetes.io/projected/b26ee112-e6d6-454e-a165-7f60ae007df9-kube-api-access-xh8lt\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:38.354594 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:38.354569 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" Oct 30 20:58:38.354789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:38.354562 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8" event={"ID":"b26ee112-e6d6-454e-a165-7f60ae007df9","Type":"ContainerDied","Data":"5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9"} Oct 30 20:58:38.354789 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:38.354706 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c818291e91650c35f9c8464adfab2fdbd1d18a9cc1a87caddbdb6de8ce0ade9" Oct 30 20:58:38.609319 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:38.609219 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:58:45.080210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.080157 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d"] Oct 30 20:58:45.080696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.080679 5855 cpu_manager.go:414] "containerMap: removing container" podUID="b26ee112-e6d6-454e-a165-7f60ae007df9" containerName="revoke-keys" Oct 30 20:58:45.080744 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.080697 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ee112-e6d6-454e-a165-7f60ae007df9" containerName="revoke-keys" Oct 30 20:58:45.080816 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.080807 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ee112-e6d6-454e-a165-7f60ae007df9" containerName="revoke-keys" Oct 30 20:58:45.084679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.084644 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.095815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.095783 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:45.095977 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.095879 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:58:45.096079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.096001 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:58:45.096079 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.096058 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:58:45.101542 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.101514 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d"] Oct 30 20:58:45.130850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.130809 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.130850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.130849 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.131121 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.131049 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxt8l\" (UniqueName: \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.232420 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.232380 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxt8l\" (UniqueName: \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.232608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.232530 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.232608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.232557 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.233341 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.233273 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.235249 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.235224 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.245780 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.245753 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxt8l\" (UniqueName: \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\") pod \"maas-api-revoke-sub-86e17370-9gb2d\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.402986 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.402888 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:45.777997 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:58:45.777958 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201f8433_1c4b_40db_b1ad_4e730947f2b5.slice/crio-eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e WatchSource:0}: Error finding container eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e: Status 404 returned error can't find the container with id eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e Oct 30 20:58:45.779773 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:45.779741 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d"] Oct 30 20:58:46.491124 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:46.491007 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" event={"ID":"201f8433-1c4b-40db-b1ad-4e730947f2b5","Type":"ContainerStarted","Data":"3bd89aaa889932b7996ec2c7fb73ea60c9948fc7e56702bbd0f613a4abd96ec5"} Oct 30 20:58:46.491511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:46.491138 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" event={"ID":"201f8433-1c4b-40db-b1ad-4e730947f2b5","Type":"ContainerStarted","Data":"eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e"} Oct 30 20:58:46.514720 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:46.514644 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" podStartSLOduration=1.5146268649999999 podStartE2EDuration="1.514626865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:46.509618844 +0000 UTC m=+1139.868331583" watchObservedRunningTime="2026-10-30 20:58:46.514626865 +0000 UTC m=+1139.873339632" Oct 30 20:58:47.503442 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:47.503403 5855 generic.go:356] "Generic (PLEG): container finished" podID="201f8433-1c4b-40db-b1ad-4e730947f2b5" containerID="3bd89aaa889932b7996ec2c7fb73ea60c9948fc7e56702bbd0f613a4abd96ec5" exitCode=0 Oct 30 20:58:47.503942 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:47.503553 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" event={"ID":"201f8433-1c4b-40db-b1ad-4e730947f2b5","Type":"ContainerDied","Data":"3bd89aaa889932b7996ec2c7fb73ea60c9948fc7e56702bbd0f613a4abd96ec5"} Oct 30 20:58:48.681822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.681789 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:48.824668 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.816922 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\") pod \"201f8433-1c4b-40db-b1ad-4e730947f2b5\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " Oct 30 20:58:48.824668 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.824604 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\" (UniqueName: \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\") pod \"201f8433-1c4b-40db-b1ad-4e730947f2b5\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " Oct 30 20:58:48.824946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.824798 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\") pod \"201f8433-1c4b-40db-b1ad-4e730947f2b5\" (UID: \"201f8433-1c4b-40db-b1ad-4e730947f2b5\") " Oct 30 20:58:48.830835 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.830786 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca" pod "201f8433-1c4b-40db-b1ad-4e730947f2b5" (UID: "201f8433-1c4b-40db-b1ad-4e730947f2b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:48.850866 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.842618 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca" pod "201f8433-1c4b-40db-b1ad-4e730947f2b5" (UID: "201f8433-1c4b-40db-b1ad-4e730947f2b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:48.856316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.856261 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l" pod "201f8433-1c4b-40db-b1ad-4e730947f2b5" (UID: "201f8433-1c4b-40db-b1ad-4e730947f2b5"). InnerVolumeSpecName "kube-api-access-rxt8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:48.932130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.932092 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.932130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.932121 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxt8l\" (UniqueName: \"kubernetes.io/projected/201f8433-1c4b-40db-b1ad-4e730947f2b5-kube-api-access-rxt8l\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.932130 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:48.932132 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f8433-1c4b-40db-b1ad-4e730947f2b5-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:49.141471 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.141371 5855 scope.go:122] "RemoveContainer" containerID="28379411682ab85a7ce0450915cda727947aa8d311dce5ca718ee05868ab9eed" Oct 30 20:58:49.153768 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.153734 5855 scope.go:122] "RemoveContainer" containerID="84951baa67eedc31b8095ecf087599dfbaac7a7ffa9266905a0b43d3c9054995" Oct 30 20:58:49.165083 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.165056 5855 scope.go:122] "RemoveContainer" containerID="51deae9fa1a337aa2eebeaafac427b9b63cf641a401dd6cdd7aa20d3032f9edd" Oct 30 20:58:49.176328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.176298 5855 scope.go:122] "RemoveContainer" containerID="a901405d811c6aedc4b9ccbfd7f07e7736c1768000eec91fdbc61c6ebb194db6" Oct 30 20:58:49.196711 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.196640 5855 scope.go:122] "RemoveContainer" containerID="fdec84f6e04ff76fbc204e73e53627812d07d9a39acedd7eacb5ef601fd59f03" Oct 30 20:58:49.209211 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.209175 5855 scope.go:122] "RemoveContainer" containerID="84da9ca87958439ed25938ad2eeb4fdff78ef02a6c72c33a61503d68ffe70c21" Oct 30 20:58:49.220460 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.220435 5855 scope.go:122] "RemoveContainer" containerID="f6b042b6d62f7770cb9976ca7300b64d0dfce5ad58c87513673c72395a4f31ef" Oct 30 20:58:49.231293 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.231273 5855 scope.go:122] "RemoveContainer" containerID="c48980d1abad6666d5e307806aba532fef77d0636081cd8a4177d610a75d33d6" Oct 30 20:58:49.244291 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.244262 5855 scope.go:122] "RemoveContainer" containerID="ccc88e52898e2371c698287b024a1d5e8e9034b13ea39691529ee71c7d25ea7e" Oct 30 20:58:49.256027 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.256005 5855 scope.go:122] "RemoveContainer" containerID="f3cd4faf89def1a096f0abaecf2636bc38dd8dc59c4cfbc54beb1e33f236ecb6" Oct 30 20:58:49.266280 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.266256 5855 scope.go:122] "RemoveContainer" containerID="214ddfaa6548246c91f493cbe96ae28f3ed8cd39161a67734315eaf916c723df" Oct 30 20:58:49.276418 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.276389 5855 scope.go:122] "RemoveContainer" containerID="59771278ba55919e2495d40df156bfd395f7c3c1b53113a857117186ac73350f" Oct 30 20:58:49.287536 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.287494 5855 scope.go:122] "RemoveContainer" containerID="dd0ce5ae8d43623ac4e7e3cb1531218534d9a982209893828c67f21441780643" Oct 30 20:58:49.301444 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.301423 5855 scope.go:122] "RemoveContainer" containerID="007a53ec3faaad22e41bb3798bfe0bfa32b9eeb714daef296c06dc93d2f2e14c" Oct 30 20:58:49.585700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.585324 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" Oct 30 20:58:49.585700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.585323 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d" event={"ID":"201f8433-1c4b-40db-b1ad-4e730947f2b5","Type":"ContainerDied","Data":"eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e"} Oct 30 20:58:49.585700 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:49.585420 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb1a06dff8cc86a86a7ab84debb6085641a5cdbc899305bb73aa35b628bf724e" Oct 30 20:58:53.634774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:53.634712 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" containerID="cri-o://9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0" gracePeriod=45 Oct 30 20:58:53.910511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:53.910480 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:58:54.014492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014451 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014518 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014585 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014617 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014726 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014700 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\" (UniqueName: \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014838 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\") pod \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\" (UID: \"57cf0bfd-a20c-4aef-bf46-096da4c9dcff\") " Oct 30 20:58:54.014993 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.014920 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.015294 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.015265 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.016042 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.015989 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.017351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.017223 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.017714 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.017694 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.017913 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.017893 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.018590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.018557 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:54.023875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.021374 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.023875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.022380 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh" pod "57cf0bfd-a20c-4aef-bf46-096da4c9dcff" (UID: "57cf0bfd-a20c-4aef-bf46-096da4c9dcff"). InnerVolumeSpecName "kube-api-access-ctmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:54.119610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.119568 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.119610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.119609 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmgh\" (UniqueName: \"kubernetes.io/projected/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-kube-api-access-ctmgh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.119610 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.119619 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57cf0bfd-a20c-4aef-bf46-096da4c9dcff-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.631228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.631191 5855 generic.go:356] "Generic (PLEG): container finished" podID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerID="9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0" exitCode=0 Oct 30 20:58:54.631228 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.631238 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" Oct 30 20:58:54.631445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.631283 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" event={"ID":"57cf0bfd-a20c-4aef-bf46-096da4c9dcff","Type":"ContainerDied","Data":"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0"} Oct 30 20:58:54.631445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.631341 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" event={"ID":"57cf0bfd-a20c-4aef-bf46-096da4c9dcff","Type":"ContainerDied","Data":"f55ace2127e525653f26a43ba8927e6588ad061dfcf4d5483b187d8d81a23610"} Oct 30 20:58:54.631445 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.631365 5855 scope.go:122] "RemoveContainer" containerID="9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0" Oct 30 20:58:54.643325 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.643211 5855 scope.go:122] "RemoveContainer" containerID="9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0" Oct 30 20:58:54.643599 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:58:54.643553 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0\": container with ID starting with 9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0 not found: ID does not exist" containerID="9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0" Oct 30 20:58:54.643672 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.643590 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0"} err="failed to get container status \"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0\": rpc error: code = NotFound desc = could not find container \"9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0\": container with ID starting with 9363151902f76e82ef0494ed901e2b093b2afbd58967a183834443f8ece94eb0 not found: ID does not exist" Oct 30 20:58:54.666373 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.666341 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:58:54.670057 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.670031 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5"] Oct 30 20:58:54.782120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.782084 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" path="/var/lib/kubelet/pods/57cf0bfd-a20c-4aef-bf46-096da4c9dcff/volumes" Oct 30 20:58:54.852212 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:54.852170 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/simulator-w5-14cca2-kserve-55749b4f64-mgdz5" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:55.045297 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:55.045270 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd"] Oct 30 20:58:55.048877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:55.048856 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aa2339f-82tdd"] Oct 30 20:58:56.779602 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:56.779567 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c515b39-3fd8-4acf-9317-1689fc97d88f" path="/var/lib/kubelet/pods/2c515b39-3fd8-4acf-9317-1689fc97d88f/volumes" Oct 30 20:58:58.026590 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:58.026558 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc"] Oct 30 20:58:58.030198 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:58.030176 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd40a544-kp9tc"] Oct 30 20:58:58.779611 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:58:58.779575 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f76e035-c571-41e6-a7a1-c2a5f251ef4b" path="/var/lib/kubelet/pods/0f76e035-c571-41e6-a7a1-c2a5f251ef4b/volumes" Oct 30 20:59:01.058691 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:01.056357 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:59:01.268979 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:01.268906 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8"] Oct 30 20:59:01.275679 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:01.272019 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-14cca2-7w9w8"] Oct 30 20:59:01.302876 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:01.302820 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:59:02.788607 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:02.788570 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26ee112-e6d6-454e-a165-7f60ae007df9" path="/var/lib/kubelet/pods/b26ee112-e6d6-454e-a165-7f60ae007df9/volumes" Oct 30 20:59:06.035682 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:06.032554 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6"] Oct 30 20:59:06.047085 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:06.045807 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d107461b-z48l6"] Oct 30 20:59:06.780270 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:06.780235 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b87802-7223-4d66-b819-ce8f0ee5279d" path="/var/lib/kubelet/pods/25b87802-7223-4d66-b819-ce8f0ee5279d/volumes" Oct 30 20:59:09.035623 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.035579 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8"] Oct 30 20:59:09.045325 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.045279 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e4b13ce-cmqp8"] Oct 30 20:59:09.125479 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.125408 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:59:09.126051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.126017 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" containerID="cri-o://4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d" gracePeriod=30 Oct 30 20:59:09.150458 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.150388 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:59:09.150776 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.150743 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" containerID="cri-o://34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac" gracePeriod=30 Oct 30 20:59:09.158579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.158539 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:59:09.159092 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.159052 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" containerID="cri-o://b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1" gracePeriod=30 Oct 30 20:59:09.165492 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.165457 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:59:09.165856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:09.165823 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" containerID="cri-o://1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3" gracePeriod=30 Oct 30 20:59:10.779452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:10.779415 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f06a27-247f-4f12-b59d-4ce04aa60ae5" path="/var/lib/kubelet/pods/d3f06a27-247f-4f12-b59d-4ce04aa60ae5/volumes" Oct 30 20:59:12.633158 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.633113 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:12.633585 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.633181 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:12.866608 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.866569 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:12.866815 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.866633 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:12.978554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.978461 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:12.978554 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:12.978527 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:13.305368 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:13.305325 5855 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:13.305566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:13.305395 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:14.383289 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.382959 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449053 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449124 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449165 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449294 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449357 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449423 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449705 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449769 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.449817 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\") pod \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\" (UID: \"ab04a06c-ca83-4bb7-adf1-3135ca2f115a\") " Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.458942 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.459235 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.459547 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.465309 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.473781 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.465570 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.475075 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.474803 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.487719 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.487617 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "kube-api-access-mnqhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.489009 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.488941 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.489352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.489317 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token" pod "ab04a06c-ca83-4bb7-adf1-3135ca2f115a" (UID: "ab04a06c-ca83-4bb7-adf1-3135ca2f115a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551687 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551723 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551738 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551754 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551768 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551782 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551795 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551809 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqhf\" (UniqueName: \"kubernetes.io/projected/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-kube-api-access-mnqhf\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.551850 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.551825 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab04a06c-ca83-4bb7-adf1-3135ca2f115a-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.566627 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.566593 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:59:14.649895 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.649864 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:59:14.652875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.652814 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654525 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654593 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654671 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654618 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.653505 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.654809 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654752 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654813 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654850 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654881 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.654946 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654926 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\") pod \"3d8b1f43-ed6a-4134-a242-bff6301be868\" (UID: \"3d8b1f43-ed6a-4134-a242-bff6301be868\") " Oct 30 20:59:14.655133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.654953 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.655133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655040 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.655292 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655273 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.655461 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655442 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.655601 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655583 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.655713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655608 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.655713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655623 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.655713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655637 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.655713 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.655680 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d8b1f43-ed6a-4134-a242-bff6301be868-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.657949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.657909 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "kube-api-access-xs8nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.657949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.657929 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.658133 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.658113 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.658517 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.658500 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy" pod "3d8b1f43-ed6a-4134-a242-bff6301be868" (UID: "3d8b1f43-ed6a-4134-a242-bff6301be868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.664534 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.664509 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:59:14.756761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756673 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.756761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756726 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.756761 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756752 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756813 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756872 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756876 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756940 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.756973 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757045 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757009 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757080 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757122 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757152 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757210 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757225 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757241 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757275 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757351 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757325 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757364 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757398 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757430 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\") pod \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\" (UID: \"79e4c813-9f4b-4049-b86c-d7beefe32ca5\") " Oct 30 20:59:14.757774 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757479 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\") pod \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\" (UID: \"5dc8bdd5-177f-4065-a98c-d53371cc4ab1\") " Oct 30 20:59:14.757992 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757422 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757986 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757432 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757446 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757519 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757693 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.758054 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.757846 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758152 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758171 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758188 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs8nr\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-kube-api-access-xs8nr\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758203 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758217 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758233 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758249 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758263 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758276 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758290 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758304 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758318 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758331 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.758352 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.758345 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d8b1f43-ed6a-4134-a242-bff6301be868-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.760167 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760137 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.760507 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760481 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.760785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760762 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.760785 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760779 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "kube-api-access-hdwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.760909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760796 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "kube-api-access-n5fw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.760909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760836 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy" pod "79e4c813-9f4b-4049-b86c-d7beefe32ca5" (UID: "79e4c813-9f4b-4049-b86c-d7beefe32ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.761020 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.760915 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.762139 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.762124 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo" pod "5dc8bdd5-177f-4065-a98c-d53371cc4ab1" (UID: "5dc8bdd5-177f-4065-a98c-d53371cc4ab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.859764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859718 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859764 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859761 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859777 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859794 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e4c813-9f4b-4049-b86c-d7beefe32ca5-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859808 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwr4\" (UniqueName: \"kubernetes.io/projected/79e4c813-9f4b-4049-b86c-d7beefe32ca5-kube-api-access-hdwr4\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859824 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859838 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.859978 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.859853 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fw5\" (UniqueName: \"kubernetes.io/projected/5dc8bdd5-177f-4065-a98c-d53371cc4ab1-kube-api-access-n5fw5\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.870416 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.870385 5855 generic.go:356] "Generic (PLEG): container finished" podID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerID="4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d" exitCode=0 Oct 30 20:59:14.870581 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.870430 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" Oct 30 20:59:14.870581 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.870460 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" event={"ID":"ab04a06c-ca83-4bb7-adf1-3135ca2f115a","Type":"ContainerDied","Data":"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d"} Oct 30 20:59:14.870581 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.870510 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p" event={"ID":"ab04a06c-ca83-4bb7-adf1-3135ca2f115a","Type":"ContainerDied","Data":"43ac19431f30558858922536b201c0eea96738be119b7404f94e4b13a04bb22d"} Oct 30 20:59:14.870581 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.870530 5855 scope.go:122] "RemoveContainer" containerID="4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d" Oct 30 20:59:14.872231 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.872210 5855 generic.go:356] "Generic (PLEG): container finished" podID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerID="1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3" exitCode=0 Oct 30 20:59:14.872365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.872290 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" event={"ID":"79e4c813-9f4b-4049-b86c-d7beefe32ca5","Type":"ContainerDied","Data":"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3"} Oct 30 20:59:14.872365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.872320 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" Oct 30 20:59:14.872365 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.872340 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx" event={"ID":"79e4c813-9f4b-4049-b86c-d7beefe32ca5","Type":"ContainerDied","Data":"67411d76f67adef2f0bd7ed4a4788e35a59bc1568c25a91722712dc2bce587fc"} Oct 30 20:59:14.878767 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.878738 5855 generic.go:356] "Generic (PLEG): container finished" podID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerID="34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac" exitCode=0 Oct 30 20:59:14.878921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.878871 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" event={"ID":"5dc8bdd5-177f-4065-a98c-d53371cc4ab1","Type":"ContainerDied","Data":"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac"} Oct 30 20:59:14.878921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.878883 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" Oct 30 20:59:14.878921 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.878905 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s" event={"ID":"5dc8bdd5-177f-4065-a98c-d53371cc4ab1","Type":"ContainerDied","Data":"d808344ac99628d28dbe281d4c27888cc9951d60e8440ba71def8fbd886f08fc"} Oct 30 20:59:14.880544 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.880516 5855 generic.go:356] "Generic (PLEG): container finished" podID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerID="b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1" exitCode=0 Oct 30 20:59:14.880723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.880569 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" Oct 30 20:59:14.880723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.880600 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" event={"ID":"3d8b1f43-ed6a-4134-a242-bff6301be868","Type":"ContainerDied","Data":"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1"} Oct 30 20:59:14.880723 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.880687 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl" event={"ID":"3d8b1f43-ed6a-4134-a242-bff6301be868","Type":"ContainerDied","Data":"e26f45993c4902188ca712c2aeb07843d894aeaf2a0b10ae30cc94b7b0b6e0d7"} Oct 30 20:59:14.883855 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.883840 5855 scope.go:122] "RemoveContainer" containerID="4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d" Oct 30 20:59:14.884172 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:14.884147 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d\": container with ID starting with 4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d not found: ID does not exist" containerID="4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d" Oct 30 20:59:14.884257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.884178 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d"} err="failed to get container status \"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d\": rpc error: code = NotFound desc = could not find container \"4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d\": container with ID starting with 4537596197f6bdcfbfbf2305539827d446360387a73b1a75d35a63bf55ecc80d not found: ID does not exist" Oct 30 20:59:14.884257 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.884198 5855 scope.go:122] "RemoveContainer" containerID="1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3" Oct 30 20:59:14.896360 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.896338 5855 scope.go:122] "RemoveContainer" containerID="1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3" Oct 30 20:59:14.896810 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:14.896779 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3\": container with ID starting with 1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3 not found: ID does not exist" containerID="1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3" Oct 30 20:59:14.896934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.896818 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3"} err="failed to get container status \"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3\": rpc error: code = NotFound desc = could not find container \"1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3\": container with ID starting with 1af12c9ed8ec18938b023a6a8e7a61af08b9a35e2058ba8fa457835c82125bb3 not found: ID does not exist" Oct 30 20:59:14.896934 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.896842 5855 scope.go:122] "RemoveContainer" containerID="34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac" Oct 30 20:59:14.898947 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.898920 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:59:14.902897 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.902868 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-jk8dx"] Oct 30 20:59:14.910022 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.909918 5855 scope.go:122] "RemoveContainer" containerID="34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac" Oct 30 20:59:14.910410 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:14.910371 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac\": container with ID starting with 34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac not found: ID does not exist" containerID="34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac" Oct 30 20:59:14.911452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.910415 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac"} err="failed to get container status \"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac\": rpc error: code = NotFound desc = could not find container \"34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac\": container with ID starting with 34a35168c15bd2ee8d17f0bf761426a9e77cb1456acb836bc6eb67ca7366b8ac not found: ID does not exist" Oct 30 20:59:14.911452 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.910443 5855 scope.go:122] "RemoveContainer" containerID="b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1" Oct 30 20:59:14.916177 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.916018 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:59:14.921811 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.921783 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-bjj4p"] Oct 30 20:59:14.922909 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.922889 5855 scope.go:122] "RemoveContainer" containerID="b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1" Oct 30 20:59:14.923264 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:14.923245 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1\": container with ID starting with b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1 not found: ID does not exist" containerID="b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1" Oct 30 20:59:14.923324 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.923276 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1"} err="failed to get container status \"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1\": rpc error: code = NotFound desc = could not find container \"b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1\": container with ID starting with b42e84322d75defc1e315a4e96801aaac4bde9e3c6ba79cf3ea300ef97f038f1 not found: ID does not exist" Oct 30 20:59:14.937114 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.937082 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:59:14.943874 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.943831 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k9nnl"] Oct 30 20:59:14.956667 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.956620 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:59:14.961518 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:14.961495 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-cn89s"] Oct 30 20:59:16.779511 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:16.779479 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" path="/var/lib/kubelet/pods/3d8b1f43-ed6a-4134-a242-bff6301be868/volumes" Oct 30 20:59:16.780042 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:16.780025 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" path="/var/lib/kubelet/pods/5dc8bdd5-177f-4065-a98c-d53371cc4ab1/volumes" Oct 30 20:59:16.780456 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:16.780427 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" path="/var/lib/kubelet/pods/79e4c813-9f4b-4049-b86c-d7beefe32ca5/volumes" Oct 30 20:59:16.781267 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:16.781248 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" path="/var/lib/kubelet/pods/ab04a06c-ca83-4bb7-adf1-3135ca2f115a/volumes" Oct 30 20:59:17.027128 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:17.027086 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn"] Oct 30 20:59:17.030370 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:17.030292 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde9a306-9snkn"] Oct 30 20:59:17.285623 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:17.285545 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:59:18.781103 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:18.781065 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cd4639-18bd-49b2-9e59-f4562f63df27" path="/var/lib/kubelet/pods/89cd4639-18bd-49b2-9e59-f4562f63df27/volumes" Oct 30 20:59:24.034120 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:24.034068 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72"] Oct 30 20:59:24.037210 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:24.037180 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef9f8d2-p7h72"] Oct 30 20:59:24.780394 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:24.780312 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f" path="/var/lib/kubelet/pods/c9f2862c-3758-4aa1-8bd4-ab3d2ee25e4f/volumes" Oct 30 20:59:30.925313 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:30.925269 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 20:59:33.051830 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:33.051782 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv"] Oct 30 20:59:33.063187 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:33.063028 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz"] Oct 30 20:59:33.066741 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:33.066713 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127d293-lsjzz"] Oct 30 20:59:33.069875 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:33.069855 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eee999e-fkfnv"] Oct 30 20:59:34.362956 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.362896 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2"] Oct 30 20:59:34.363724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363694 5855 cpu_manager.go:414] "containerMap: removing container" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" Oct 30 20:59:34.363724 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363720 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363738 5855 cpu_manager.go:414] "containerMap: removing container" podUID="201f8433-1c4b-40db-b1ad-4e730947f2b5" containerName="revoke-keys" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363746 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201f8433-1c4b-40db-b1ad-4e730947f2b5" containerName="revoke-keys" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363765 5855 cpu_manager.go:414] "containerMap: removing container" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363775 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363795 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363800 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363810 5855 cpu_manager.go:414] "containerMap: removing container" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363814 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363834 5855 cpu_manager.go:414] "containerMap: removing container" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" Oct 30 20:59:34.363846 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363838 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363934 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8b1f43-ed6a-4134-a242-bff6301be868" containerName="istio-proxy" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363942 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab04a06c-ca83-4bb7-adf1-3135ca2f115a" containerName="istio-proxy" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363952 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc8bdd5-177f-4065-a98c-d53371cc4ab1" containerName="istio-proxy" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363959 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e4c813-9f4b-4049-b86c-d7beefe32ca5" containerName="istio-proxy" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363964 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="201f8433-1c4b-40db-b1ad-4e730947f2b5" containerName="revoke-keys" Oct 30 20:59:34.364183 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.363969 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cf0bfd-a20c-4aef-bf46-096da4c9dcff" containerName="main" Oct 30 20:59:34.367439 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.367410 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.374497 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.374468 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.374755 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.374677 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:59:34.374838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.374777 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:59:34.374838 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.374835 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.375949 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.375920 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2"] Oct 30 20:59:34.494548 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.494505 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gjc8\" (UniqueName: \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.494777 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.494746 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.494827 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.494812 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.596046 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.596006 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gjc8\" (UniqueName: \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.596307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.596141 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.596307 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.596194 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.596962 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.596933 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.598810 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.598782 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.609033 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.609006 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gjc8\" (UniqueName: \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\") pod \"maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.686300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.686207 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:34.783254 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.782450 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606505b8-108e-47ea-8833-d5145896da78" path="/var/lib/kubelet/pods/606505b8-108e-47ea-8833-d5145896da78/volumes" Oct 30 20:59:34.783254 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.782958 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99de9906-c352-4212-b527-3f215c951208" path="/var/lib/kubelet/pods/99de9906-c352-4212-b527-3f215c951208/volumes" Oct 30 20:59:34.968024 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:34.967987 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2"] Oct 30 20:59:34.968286 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:59:34.968257 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb464cea_dc5c_4032_a4a5_cbc050df3474.slice/crio-3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b WatchSource:0}: Error finding container 3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b: Status 404 returned error can't find the container with id 3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b Oct 30 20:59:35.085248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:35.085206 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" event={"ID":"bb464cea-dc5c-4032-a4a5-cbc050df3474","Type":"ContainerStarted","Data":"3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b"} Oct 30 20:59:36.094877 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:36.094837 5855 generic.go:356] "Generic (PLEG): container finished" podID="bb464cea-dc5c-4032-a4a5-cbc050df3474" containerID="60ce51896d3becf10b4ae5ed9b548ebcf251efbb4e2363c6341791d9e937b992" exitCode=0 Oct 30 20:59:36.095330 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:36.094925 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" event={"ID":"bb464cea-dc5c-4032-a4a5-cbc050df3474","Type":"ContainerDied","Data":"60ce51896d3becf10b4ae5ed9b548ebcf251efbb4e2363c6341791d9e937b992"} Oct 30 20:59:37.308201 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.308170 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:37.457171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.457087 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\") pod \"bb464cea-dc5c-4032-a4a5-cbc050df3474\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " Oct 30 20:59:37.457171 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.457171 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\" (UniqueName: \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\") pod \"bb464cea-dc5c-4032-a4a5-cbc050df3474\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " Oct 30 20:59:37.457367 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.457195 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\") pod \"bb464cea-dc5c-4032-a4a5-cbc050df3474\" (UID: \"bb464cea-dc5c-4032-a4a5-cbc050df3474\") " Oct 30 20:59:37.457579 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.457558 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca" pod "bb464cea-dc5c-4032-a4a5-cbc050df3474" (UID: "bb464cea-dc5c-4032-a4a5-cbc050df3474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:37.459516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.459496 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8" pod "bb464cea-dc5c-4032-a4a5-cbc050df3474" (UID: "bb464cea-dc5c-4032-a4a5-cbc050df3474"). InnerVolumeSpecName "kube-api-access-7gjc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:37.459516 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.459498 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca" pod "bb464cea-dc5c-4032-a4a5-cbc050df3474" (UID: "bb464cea-dc5c-4032-a4a5-cbc050df3474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:37.558367 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.558329 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.558367 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.558365 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gjc8\" (UniqueName: \"kubernetes.io/projected/bb464cea-dc5c-4032-a4a5-cbc050df3474-kube-api-access-7gjc8\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.558591 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:37.558380 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb464cea-dc5c-4032-a4a5-cbc050df3474-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:38.147150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:38.147104 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" event={"ID":"bb464cea-dc5c-4032-a4a5-cbc050df3474","Type":"ContainerDied","Data":"3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b"} Oct 30 20:59:38.147150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:38.147146 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2" Oct 30 20:59:38.147150 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:38.147158 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3577e10a81f09a4dacf9b5b1ea4f3f3d5ee3cc761626d0291526cd98c946b34b" Oct 30 20:59:39.146766 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:39.146723 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:59:39.180899 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:39.180813 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:59:44.443096 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.443041 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" podUID="f7ada5c68d422440bbf7fd2285220de0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:44.637668 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.637615 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g"] Oct 30 20:59:44.638193 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.638176 5855 cpu_manager.go:414] "containerMap: removing container" podUID="bb464cea-dc5c-4032-a4a5-cbc050df3474" containerName="revoke-keys" Oct 30 20:59:44.638245 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.638194 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb464cea-dc5c-4032-a4a5-cbc050df3474" containerName="revoke-keys" Oct 30 20:59:44.638277 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.638268 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb464cea-dc5c-4032-a4a5-cbc050df3474" containerName="revoke-keys" Oct 30 20:59:44.642300 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.642274 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.649704 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.649673 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:59:44.649854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.649750 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:59:44.649854 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.649825 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:59:44.649936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.649864 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:59:44.649936 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.649919 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g"] Oct 30 20:59:44.844746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.842048 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdxf\" (UniqueName: \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.844746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.842139 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.844746 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.842308 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.943519 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.943481 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.943798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.943558 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdxf\" (UniqueName: \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.943798 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.943613 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.944856 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.944828 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.946413 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.946347 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.957015 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.956992 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdxf\" (UniqueName: \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\") pod \"maas-api-revoke-sub-60515392-rz69g\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:44.962833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:44.962811 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:45.296367 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:45.296332 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g"] Oct 30 20:59:45.299887 ip-10-0-141-65 kubenswrapper[5855]: W1030 20:59:45.299859 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ea5d42_f76c_48c4_b812_57096189e0f3.slice/crio-41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c WatchSource:0}: Error finding container 41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c: Status 404 returned error can't find the container with id 41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c Oct 30 20:59:46.251174 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:46.251137 5855 generic.go:356] "Generic (PLEG): container finished" podID="46ea5d42-f76c-48c4-b812-57096189e0f3" containerID="72b4c736005dffb1fec62c8598bfd18992828712217d5ba2f63dd31671c36eda" exitCode=0 Oct 30 20:59:46.251566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:46.251223 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" event={"ID":"46ea5d42-f76c-48c4-b812-57096189e0f3","Type":"ContainerDied","Data":"72b4c736005dffb1fec62c8598bfd18992828712217d5ba2f63dd31671c36eda"} Oct 30 20:59:46.251566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:46.251311 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" event={"ID":"46ea5d42-f76c-48c4-b812-57096189e0f3","Type":"ContainerStarted","Data":"41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c"} Oct 30 20:59:47.039773 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.039722 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7"] Oct 30 20:59:47.047688 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.043582 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e0148-dfzz7"] Oct 30 20:59:47.504636 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.504601 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:47.512248 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.512218 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\" (UniqueName: \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\") pod \"46ea5d42-f76c-48c4-b812-57096189e0f3\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " Oct 30 20:59:47.512455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.512325 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\") pod \"46ea5d42-f76c-48c4-b812-57096189e0f3\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " Oct 30 20:59:47.512455 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.512407 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\") pod \"46ea5d42-f76c-48c4-b812-57096189e0f3\" (UID: \"46ea5d42-f76c-48c4-b812-57096189e0f3\") " Oct 30 20:59:47.513012 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.512981 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca" pod "46ea5d42-f76c-48c4-b812-57096189e0f3" (UID: "46ea5d42-f76c-48c4-b812-57096189e0f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:47.515134 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.515089 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf" pod "46ea5d42-f76c-48c4-b812-57096189e0f3" (UID: "46ea5d42-f76c-48c4-b812-57096189e0f3"). InnerVolumeSpecName "kube-api-access-ssdxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:47.515482 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.515457 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca" pod "46ea5d42-f76c-48c4-b812-57096189e0f3" (UID: "46ea5d42-f76c-48c4-b812-57096189e0f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:47.614230 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.614191 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:47.614230 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.614223 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46ea5d42-f76c-48c4-b812-57096189e0f3-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:47.614230 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:47.614233 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdxf\" (UniqueName: \"kubernetes.io/projected/46ea5d42-f76c-48c4-b812-57096189e0f3-kube-api-access-ssdxf\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.081737 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.076749 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6"] Oct 30 20:59:48.083620 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.083588 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt"] Oct 30 20:59:48.088945 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.088909 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f45da628-lnpt6"] Oct 30 20:59:48.094941 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.094891 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5421489-f8qlt"] Oct 30 20:59:48.268822 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.268791 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" Oct 30 20:59:48.268999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.268792 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g" event={"ID":"46ea5d42-f76c-48c4-b812-57096189e0f3","Type":"ContainerDied","Data":"41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c"} Oct 30 20:59:48.268999 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.268935 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f94b9e46309954acbaf82d1db714ddf2562720cedd58073a3978eb70edc97c" Oct 30 20:59:48.780995 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.780957 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412de240-9d58-4923-8926-6001b1d33f7d" path="/var/lib/kubelet/pods/412de240-9d58-4923-8926-6001b1d33f7d/volumes" Oct 30 20:59:48.781584 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.781565 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8310579d-fbe3-4e6d-a050-5893d470d72f" path="/var/lib/kubelet/pods/8310579d-fbe3-4e6d-a050-5893d470d72f/volumes" Oct 30 20:59:48.782165 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:48.782148 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa927e25-3d99-40a7-a06c-b04601af6192" path="/var/lib/kubelet/pods/fa927e25-3d99-40a7-a06c-b04601af6192/volumes" Oct 30 20:59:49.591837 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:49.591797 5855 scope.go:122] "RemoveContainer" containerID="9e294862d6deda734020277dcb23d8392d87c06565db21eede1c6a82d9967999" Oct 30 20:59:54.195698 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.195633 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" containerID="cri-o://87d9fc2029a000d30371dc5e446fb75854289ef2f7811a1e3a6e20d2fd5192bd" gracePeriod=45 Oct 30 20:59:54.210316 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.210255 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" containerID="cri-o://35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb" gracePeriod=45 Oct 30 20:59:54.318596 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.318547 5855 generic.go:356] "Generic (PLEG): container finished" podID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerID="87d9fc2029a000d30371dc5e446fb75854289ef2f7811a1e3a6e20d2fd5192bd" exitCode=0 Oct 30 20:59:54.318828 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.318624 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" event={"ID":"b11f0200-eede-4ff2-a1aa-8bebcb373902","Type":"ContainerDied","Data":"87d9fc2029a000d30371dc5e446fb75854289ef2f7811a1e3a6e20d2fd5192bd"} Oct 30 20:59:54.363882 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.363833 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused" Oct 30 20:59:54.513236 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.513196 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:59:54.517515 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.517454 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685080 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\" (UniqueName: \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685159 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685211 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685243 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685282 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\" (UniqueName: \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685337 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685359 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685398 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685417 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685466 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685493 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\") pod \"b11f0200-eede-4ff2-a1aa-8bebcb373902\" (UID: \"b11f0200-eede-4ff2-a1aa-8bebcb373902\") " Oct 30 20:59:54.685707 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.685525 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\" (UniqueName: \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\") pod \"e5da1eae-02ea-476c-b086-247197542e8f\" (UID: \"e5da1eae-02ea-476c-b086-247197542e8f\") " Oct 30 20:59:54.686583 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.686551 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.688077 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.687864 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.688710 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.688321 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.689192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.688912 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.689457 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.689427 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.695903 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.689835 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.722196 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.722153 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:54.734833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.726231 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.734833 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.726429 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.741515 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.741463 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "kube-api-access-78mts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:54.753823 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.753767 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr" pod "b11f0200-eede-4ff2-a1aa-8bebcb373902" (UID: "b11f0200-eede-4ff2-a1aa-8bebcb373902"). InnerVolumeSpecName "kube-api-access-5trdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:54.758439 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.758396 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs" pod "e5da1eae-02ea-476c-b086-247197542e8f" (UID: "e5da1eae-02ea-476c-b086-247197542e8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:54.787023 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.786991 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5da1eae-02ea-476c-b086-247197542e8f-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787023 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787023 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787035 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787048 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787060 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78mts\" (UniqueName: \"kubernetes.io/projected/e5da1eae-02ea-476c-b086-247197542e8f-kube-api-access-78mts\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787072 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b11f0200-eede-4ff2-a1aa-8bebcb373902-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787083 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787093 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787104 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787115 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5trdr\" (UniqueName: \"kubernetes.io/projected/b11f0200-eede-4ff2-a1aa-8bebcb373902-kube-api-access-5trdr\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787127 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5da1eae-02ea-476c-b086-247197542e8f-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.787256 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:54.787139 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b11f0200-eede-4ff2-a1aa-8bebcb373902-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:55.328366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.328333 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" Oct 30 20:59:55.328366 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.328347 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf" event={"ID":"b11f0200-eede-4ff2-a1aa-8bebcb373902","Type":"ContainerDied","Data":"48a1beee486efffa1dd791ec2561e5987bca42c7bd2bb6b4dfa3a1b458bc06cc"} Oct 30 20:59:55.329031 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.328403 5855 scope.go:122] "RemoveContainer" containerID="87d9fc2029a000d30371dc5e446fb75854289ef2f7811a1e3a6e20d2fd5192bd" Oct 30 20:59:55.330058 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.330034 5855 generic.go:356] "Generic (PLEG): container finished" podID="e5da1eae-02ea-476c-b086-247197542e8f" containerID="35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb" exitCode=0 Oct 30 20:59:55.330138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.330084 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" Oct 30 20:59:55.330138 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.330120 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" event={"ID":"e5da1eae-02ea-476c-b086-247197542e8f","Type":"ContainerDied","Data":"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb"} Oct 30 20:59:55.330235 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.330158 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp" event={"ID":"e5da1eae-02ea-476c-b086-247197542e8f","Type":"ContainerDied","Data":"3dc2f6d2983428031e8663c5e5bf0bd7f3c5da89a20d49b613a980bf23caf124"} Oct 30 20:59:55.341192 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.341170 5855 scope.go:122] "RemoveContainer" containerID="35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb" Oct 30 20:59:55.353335 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.353303 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:59:55.354566 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.354547 5855 scope.go:122] "RemoveContainer" containerID="35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb" Oct 30 20:59:55.354960 ip-10-0-141-65 kubenswrapper[5855]: E1030 20:59:55.354936 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb\": container with ID starting with 35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb not found: ID does not exist" containerID="35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb" Oct 30 20:59:55.355051 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.354976 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb"} err="failed to get container status \"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb\": rpc error: code = NotFound desc = could not find container \"35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb\": container with ID starting with 35e0fa11ae9a63b40e088e33c138bc4d6d3543405209476f435b49f38951fccb not found: ID does not exist" Oct 30 20:59:55.356546 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.356524 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/simulator-w6-3d909d-kserve-b494fdb44-pd9cp"] Oct 30 20:59:55.365540 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.365501 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:59:55.373328 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:55.373290 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-w6-3d909d-kserve-5cb87689fb-b5thf"] Oct 30 20:59:56.781227 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:56.781193 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" path="/var/lib/kubelet/pods/b11f0200-eede-4ff2-a1aa-8bebcb373902/volumes" Oct 30 20:59:56.781637 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:56.781622 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5da1eae-02ea-476c-b086-247197542e8f" path="/var/lib/kubelet/pods/e5da1eae-02ea-476c-b086-247197542e8f/volumes" Oct 30 20:59:57.879233 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:57.879194 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 20:59:58.551598 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:58.551558 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2"] Oct 30 20:59:58.555696 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:58.555643 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-3d909d-4fcc2"] Oct 30 20:59:58.856104 ip-10-0-141-65 kubenswrapper[5855]: I1030 20:59:58.852078 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb464cea-dc5c-4032-a4a5-cbc050df3474" path="/var/lib/kubelet/pods/bb464cea-dc5c-4032-a4a5-cbc050df3474/volumes" Oct 30 21:00:00.153019 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.152982 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:00.153474 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153434 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" Oct 30 21:00:00.153474 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153445 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" Oct 30 21:00:00.153474 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153474 5855 cpu_manager.go:414] "containerMap: removing container" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153480 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153495 5855 cpu_manager.go:414] "containerMap: removing container" podUID="46ea5d42-f76c-48c4-b812-57096189e0f3" containerName="revoke-keys" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153500 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ea5d42-f76c-48c4-b812-57096189e0f3" containerName="revoke-keys" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153566 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ea5d42-f76c-48c4-b812-57096189e0f3" containerName="revoke-keys" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153575 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5da1eae-02ea-476c-b086-247197542e8f" containerName="main" Oct 30 21:00:00.153587 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.153584 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11f0200-eede-4ff2-a1aa-8bebcb373902" containerName="main" Oct 30 21:00:00.158367 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.158339 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.159343 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.159314 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:00.163117 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.163094 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.164228 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.163916 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:00:00.165631 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.165599 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:00:00.165631 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.165611 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:00:00.190989 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.190944 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:00.250715 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.238460 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:00.262353 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.262315 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxqn\" (UniqueName: \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.270685 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.262637 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm542\" (UniqueName: \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.282018 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.271064 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.282828 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.282778 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.384280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.384232 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.384506 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.384346 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxqn\" (UniqueName: \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.384506 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.384372 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm542\" (UniqueName: \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.384506 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.384470 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.385171 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.385147 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.385230 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.385172 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.397578 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.397538 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm542\" (UniqueName: \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\") pod \"maas-api-key-cleanup-29889900-hp74v\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.397792 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.397705 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxqn\" (UniqueName: \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\") pod \"maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.526177 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.526082 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:00.613174 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.608970 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:00.818004 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:00:00.817969 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a811d8_1441_4211_8714_cc4a5eecfa3e.slice/crio-00fe7842ccb3ed1540ddbb17a0a3eee6ce32228f49d5fd0dc1094286b3a2a659 WatchSource:0}: Error finding container 00fe7842ccb3ed1540ddbb17a0a3eee6ce32228f49d5fd0dc1094286b3a2a659: Status 404 returned error can't find the container with id 00fe7842ccb3ed1540ddbb17a0a3eee6ce32228f49d5fd0dc1094286b3a2a659 Oct 30 21:00:00.819800 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.819773 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:00.933625 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:00.933420 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:00.948272 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:00.948237 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:00:01.386770 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.382987 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerStarted","Data":"3f712dea49ec67dc0b565e71c2a53504af87797785bf75203812554b0969664f"} Oct 30 21:00:01.386770 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.383090 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerStarted","Data":"85228eaf43520f4ac107241492cc4e7ad0fb4fa8a99a30c71eb450fd26745b88"} Oct 30 21:00:01.436401 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.436326 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" podStartSLOduration=1.436302194 podStartE2EDuration="1.436302194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.415965872 +0000 UTC m=+1214.774678645" watchObservedRunningTime="2026-10-30 21:00:01.436302194 +0000 UTC m=+1214.795014942" Oct 30 21:00:01.470684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.462121 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerStarted","Data":"221c9d030744bf9fe4393061cbe0b735718d3f8199e2ea6c6be63e17d671c329"} Oct 30 21:00:01.470684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.462202 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerStarted","Data":"00fe7842ccb3ed1540ddbb17a0a3eee6ce32228f49d5fd0dc1094286b3a2a659"} Oct 30 21:00:01.506071 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:01.501473 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" podStartSLOduration=1.5014528139999999 podStartE2EDuration="1.501452814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.50001894 +0000 UTC m=+1214.858731687" watchObservedRunningTime="2026-10-30 21:00:01.501452814 +0000 UTC m=+1214.860165560" Oct 30 21:00:02.490079 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.490039 5855 generic.go:356] "Generic (PLEG): container finished" podID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerID="3f712dea49ec67dc0b565e71c2a53504af87797785bf75203812554b0969664f" exitCode=60 Oct 30 21:00:02.490718 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.490111 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerDied","Data":"3f712dea49ec67dc0b565e71c2a53504af87797785bf75203812554b0969664f"} Oct 30 21:00:02.490718 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.490556 5855 scope.go:122] "RemoveContainer" containerID="3f712dea49ec67dc0b565e71c2a53504af87797785bf75203812554b0969664f" Oct 30 21:00:02.492448 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.492368 5855 generic.go:356] "Generic (PLEG): container finished" podID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerID="221c9d030744bf9fe4393061cbe0b735718d3f8199e2ea6c6be63e17d671c329" exitCode=60 Oct 30 21:00:02.492533 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.492456 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerDied","Data":"221c9d030744bf9fe4393061cbe0b735718d3f8199e2ea6c6be63e17d671c329"} Oct 30 21:00:02.492764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:02.492745 5855 scope.go:122] "RemoveContainer" containerID="221c9d030744bf9fe4393061cbe0b735718d3f8199e2ea6c6be63e17d671c329" Oct 30 21:00:03.552680 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:03.544273 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerStarted","Data":"858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a"} Oct 30 21:00:03.579742 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:03.577631 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerStarted","Data":"0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144"} Oct 30 21:00:04.590358 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.590325 5855 generic.go:356] "Generic (PLEG): container finished" podID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerID="858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a" exitCode=60 Oct 30 21:00:04.590877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.590413 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerDied","Data":"858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a"} Oct 30 21:00:04.590877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.590478 5855 scope.go:122] "RemoveContainer" containerID="3f712dea49ec67dc0b565e71c2a53504af87797785bf75203812554b0969664f" Oct 30 21:00:04.590877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.590704 5855 scope.go:122] "RemoveContainer" containerID="858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a" Oct 30 21:00:04.591026 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:04.591003 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq_odh-ai-gateway-infra(3c94b98b-e7ee-44d9-8df3-86601cfa6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" Oct 30 21:00:04.592545 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.592520 5855 generic.go:356] "Generic (PLEG): container finished" podID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerID="0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144" exitCode=60 Oct 30 21:00:04.592645 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.592598 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerDied","Data":"0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144"} Oct 30 21:00:04.592842 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.592829 5855 scope.go:122] "RemoveContainer" containerID="0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144" Oct 30 21:00:04.593076 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:04.593061 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-hp74v_odh-ai-gateway-infra(78a811d8-1441-4211-8714-cc4a5eecfa3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" Oct 30 21:00:04.603403 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:04.603384 5855 scope.go:122] "RemoveContainer" containerID="221c9d030744bf9fe4393061cbe0b735718d3f8199e2ea6c6be63e17d671c329" Oct 30 21:00:05.651799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:05.651767 5855 scope.go:122] "RemoveContainer" containerID="0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144" Oct 30 21:00:05.652265 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:05.652119 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-hp74v_odh-ai-gateway-infra(78a811d8-1441-4211-8714-cc4a5eecfa3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" Oct 30 21:00:05.657728 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:05.657683 5855 scope.go:122] "RemoveContainer" containerID="858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a" Oct 30 21:00:05.658065 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:05.658038 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq_odh-ai-gateway-infra(3c94b98b-e7ee-44d9-8df3-86601cfa6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" Oct 30 21:00:06.348014 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.347966 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 21:00:06.348669 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.348610 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" containerID="cri-o://7f3f861b0924688e616fd3eb92163baf75baae4a692371ddbce5763681374f12" gracePeriod=30 Oct 30 21:00:06.357795 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.356006 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 21:00:06.358304 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.358261 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" containerID="cri-o://daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858" gracePeriod=30 Oct 30 21:00:06.373228 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.373185 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 21:00:06.374934 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:06.374891 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" containerID="cri-o://b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07" gracePeriod=30 Oct 30 21:00:08.026443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.026382 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj"] Oct 30 21:00:08.029952 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.029925 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn"] Oct 30 21:00:08.032733 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.032713 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-971d237a-xszmj"] Oct 30 21:00:08.035150 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.035127 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36101384-mggkn"] Oct 30 21:00:08.781032 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.780994 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cca3ea5-6879-4af0-a616-9f678f01e951" path="/var/lib/kubelet/pods/0cca3ea5-6879-4af0-a616-9f678f01e951/volumes" Oct 30 21:00:08.781689 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:08.781628 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea05bb9-9243-408c-8a9a-fa3cd1600892" path="/var/lib/kubelet/pods/bea05bb9-9243-408c-8a9a-fa3cd1600892/volumes" Oct 30 21:00:09.675828 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:09.675785 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:00:11.518219 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.518184 5855 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:00:11.518704 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.518252 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:00:11.654426 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.654394 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 21:00:11.698892 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.698861 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 21:00:11.712121 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.712059 5855 generic.go:356] "Generic (PLEG): container finished" podID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerID="b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07" exitCode=0 Oct 30 21:00:11.712362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.712178 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" Oct 30 21:00:11.712362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.712205 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" event={"ID":"e15834ab-9d48-49b1-b6be-98655a5ec37c","Type":"ContainerDied","Data":"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07"} Oct 30 21:00:11.712362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.712260 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c" event={"ID":"e15834ab-9d48-49b1-b6be-98655a5ec37c","Type":"ContainerDied","Data":"2faa7678b3a9922701d01675d361f165df641a644eb0e09f216e6bef6803879e"} Oct 30 21:00:11.712362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.712288 5855 scope.go:122] "RemoveContainer" containerID="b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747435 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747505 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747563 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747597 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747700 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747768 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747851 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747918 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.747961 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\") pod \"70309972-d5d4-48c2-9c39-8c76e633ad3c\" (UID: \"70309972-d5d4-48c2-9c39-8c76e633ad3c\") " Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.753948 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.756920 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.757428 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.757496 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.757547 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.779691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.767815 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:11.803691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.802893 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.803691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.802913 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "kube-api-access-tj5mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.803931 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.803803 5855 generic.go:356] "Generic (PLEG): container finished" podID="1b0bf314-5df2-425e-9bee-131a00148753" containerID="7f3f861b0924688e616fd3eb92163baf75baae4a692371ddbce5763681374f12" exitCode=0 Oct 30 21:00:11.803931 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.803886 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" event={"ID":"1b0bf314-5df2-425e-9bee-131a00148753","Type":"ContainerDied","Data":"7f3f861b0924688e616fd3eb92163baf75baae4a692371ddbce5763681374f12"} Oct 30 21:00:11.810709 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.805228 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy" pod "70309972-d5d4-48c2-9c39-8c76e633ad3c" (UID: "70309972-d5d4-48c2-9c39-8c76e633ad3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.810709 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.806535 5855 generic.go:356] "Generic (PLEG): container finished" podID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerID="daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858" exitCode=0 Oct 30 21:00:11.810709 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.806648 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" event={"ID":"70309972-d5d4-48c2-9c39-8c76e633ad3c","Type":"ContainerDied","Data":"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858"} Oct 30 21:00:11.810709 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.806822 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" Oct 30 21:00:11.820074 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.819964 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q" event={"ID":"70309972-d5d4-48c2-9c39-8c76e633ad3c","Type":"ContainerDied","Data":"72eba2673e5db893ce70266d60e95062b5c65d610ee1a4603e46adb7fdf7db1e"} Oct 30 21:00:11.821757 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.821729 5855 scope.go:122] "RemoveContainer" containerID="b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07" Oct 30 21:00:11.822114 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:11.822087 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07\": container with ID starting with b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07 not found: ID does not exist" containerID="b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07" Oct 30 21:00:11.822259 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.822129 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07"} err="failed to get container status \"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07\": rpc error: code = NotFound desc = could not find container \"b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07\": container with ID starting with b51a071dd82c4fefd8a5ba4f113d5efe46fa47c9a83390077faebf3514170f07 not found: ID does not exist" Oct 30 21:00:11.822259 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.822152 5855 scope.go:122] "RemoveContainer" containerID="daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858" Oct 30 21:00:11.836491 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.836469 5855 scope.go:122] "RemoveContainer" containerID="daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858" Oct 30 21:00:11.836886 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:11.836864 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858\": container with ID starting with daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858 not found: ID does not exist" containerID="daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858" Oct 30 21:00:11.836999 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.836894 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858"} err="failed to get container status \"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858\": rpc error: code = NotFound desc = could not find container \"daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858\": container with ID starting with daabaeff184569966880cccb01836ff0695e47cdc33f30e2ba068255bedd8858 not found: ID does not exist" Oct 30 21:00:11.849254 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849234 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849336 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849265 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849336 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849297 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849336 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849325 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849483 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849352 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849483 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849459 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849682 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849533 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849682 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849593 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.849682 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849606 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.850169 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849688 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\") pod \"e15834ab-9d48-49b1-b6be-98655a5ec37c\" (UID: \"e15834ab-9d48-49b1-b6be-98655a5ec37c\") " Oct 30 21:00:11.850169 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.849809 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.850169 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850004 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.850169 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850086 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850255 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850276 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850286 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850294 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850303 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e15834ab-9d48-49b1-b6be-98655a5ec37c-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850315 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850352 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850364 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70309972-d5d4-48c2-9c39-8c76e633ad3c-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850378 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5mj\" (UniqueName: \"kubernetes.io/projected/70309972-d5d4-48c2-9c39-8c76e633ad3c-kube-api-access-tj5mj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850392 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70309972-d5d4-48c2-9c39-8c76e633ad3c-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850395 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850405 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850417 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.850444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.850426 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70309972-d5d4-48c2-9c39-8c76e633ad3c-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.851792 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.851757 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.851915 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.851901 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "kube-api-access-rc4f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.852087 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.852072 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:11.852281 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.852269 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy" pod "e15834ab-9d48-49b1-b6be-98655a5ec37c" (UID: "e15834ab-9d48-49b1-b6be-98655a5ec37c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.877057 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.877030 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 21:00:11.899977 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.899942 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-t2c2q"] Oct 30 21:00:11.951738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.951701 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.951738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.951738 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.952024 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.951756 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.952024 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.951771 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc4f7\" (UniqueName: \"kubernetes.io/projected/e15834ab-9d48-49b1-b6be-98655a5ec37c-kube-api-access-rc4f7\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.952024 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:11.951785 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e15834ab-9d48-49b1-b6be-98655a5ec37c-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.038061 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.038030 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 21:00:12.043525 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.043491 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 21:00:12.047210 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.047185 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-q7f8c"] Oct 30 21:00:12.153388 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153296 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153388 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153338 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153388 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153379 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153646 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153430 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153646 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153467 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153646 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153525 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153646 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153588 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153646 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153641 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153944 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153714 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\") pod \"1b0bf314-5df2-425e-9bee-131a00148753\" (UID: \"1b0bf314-5df2-425e-9bee-131a00148753\") " Oct 30 21:00:12.153944 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153722 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.153944 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153753 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.154094 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153946 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.154094 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153959 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.154094 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.153980 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:12.154291 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.154267 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b0bf314-5df2-425e-9bee-131a00148753-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.154373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.154297 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.154373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.154313 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.154373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.154326 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.154373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.154340 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.156113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.156084 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.156113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.156093 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "kube-api-access-vfmz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.156266 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.156141 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:12.156266 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.156157 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token" pod "1b0bf314-5df2-425e-9bee-131a00148753" (UID: "1b0bf314-5df2-425e-9bee-131a00148753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.255265 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.255225 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b0bf314-5df2-425e-9bee-131a00148753-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.255265 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.255257 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b0bf314-5df2-425e-9bee-131a00148753-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.255265 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.255272 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.255510 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.255285 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmz8\" (UniqueName: \"kubernetes.io/projected/1b0bf314-5df2-425e-9bee-131a00148753-kube-api-access-vfmz8\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.780603 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.780571 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" path="/var/lib/kubelet/pods/70309972-d5d4-48c2-9c39-8c76e633ad3c/volumes" Oct 30 21:00:12.781074 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.781060 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" path="/var/lib/kubelet/pods/e15834ab-9d48-49b1-b6be-98655a5ec37c/volumes" Oct 30 21:00:12.818182 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.818149 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" Oct 30 21:00:12.818311 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.818197 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk" event={"ID":"1b0bf314-5df2-425e-9bee-131a00148753","Type":"ContainerDied","Data":"a0bf08e9bc321bd5e358321e5b6e0ade5e5348bfebcd4839c45e95fdd417dba3"} Oct 30 21:00:12.818311 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.818261 5855 scope.go:122] "RemoveContainer" containerID="7f3f861b0924688e616fd3eb92163baf75baae4a692371ddbce5763681374f12" Oct 30 21:00:12.838742 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.838702 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 21:00:12.845216 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:12.845193 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-j6wwk"] Oct 30 21:00:13.772501 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:13.772472 5855 scope.go:122] "RemoveContainer" containerID="858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a" Oct 30 21:00:14.800681 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:14.794945 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0bf314-5df2-425e-9bee-131a00148753" path="/var/lib/kubelet/pods/1b0bf314-5df2-425e-9bee-131a00148753/volumes" Oct 30 21:00:14.845310 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:14.845218 5855 generic.go:356] "Generic (PLEG): container finished" podID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerID="3b2599963bd928b64ffc579c59307a5238f9731ee208d885f08053869baf4622" exitCode=60 Oct 30 21:00:14.845310 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:14.845288 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerDied","Data":"3b2599963bd928b64ffc579c59307a5238f9731ee208d885f08053869baf4622"} Oct 30 21:00:14.845532 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:14.845357 5855 scope.go:122] "RemoveContainer" containerID="858e4f64c38638e5202247e309c51ec4651ce77f131c70858e81ee866213eb1a" Oct 30 21:00:14.845845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:14.845827 5855 scope.go:122] "RemoveContainer" containerID="3b2599963bd928b64ffc579c59307a5238f9731ee208d885f08053869baf4622" Oct 30 21:00:14.846191 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:14.846160 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq_odh-ai-gateway-infra(3c94b98b-e7ee-44d9-8df3-86601cfa6f3f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" Oct 30 21:00:15.085427 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:15.085387 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5"] Oct 30 21:00:15.093637 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:15.093604 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c399b28-v8bj5"] Oct 30 21:00:15.880263 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:15.880225 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:16.041022 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.040995 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:16.210808 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.210718 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\" (UniqueName: \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\") pod \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " Oct 30 21:00:16.210972 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.210852 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\") pod \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\" (UID: \"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f\") " Oct 30 21:00:16.211236 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.211212 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca" pod "3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" (UID: "3c94b98b-e7ee-44d9-8df3-86601cfa6f3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:16.213314 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.213290 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn" pod "3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" (UID: "3c94b98b-e7ee-44d9-8df3-86601cfa6f3f"). InnerVolumeSpecName "kube-api-access-9hxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:16.312464 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.312432 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxqn\" (UniqueName: \"kubernetes.io/projected/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-kube-api-access-9hxqn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:16.312464 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.312461 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f-maas-api-service-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:16.780638 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.780603 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534a4532-6a64-4875-bfa9-58a7ed15edc9" path="/var/lib/kubelet/pods/534a4532-6a64-4875-bfa9-58a7ed15edc9/volumes" Oct 30 21:00:16.862922 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.862894 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" Oct 30 21:00:16.863097 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.862934 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq" event={"ID":"3c94b98b-e7ee-44d9-8df3-86601cfa6f3f","Type":"ContainerDied","Data":"85228eaf43520f4ac107241492cc4e7ad0fb4fa8a99a30c71eb450fd26745b88"} Oct 30 21:00:16.863097 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.863000 5855 scope.go:122] "RemoveContainer" containerID="3b2599963bd928b64ffc579c59307a5238f9731ee208d885f08053869baf4622" Oct 30 21:00:16.883009 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.882978 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:16.887124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:16.887098 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b8aee1-29889900-bc6rq"] Oct 30 21:00:17.772326 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:17.772296 5855 scope.go:122] "RemoveContainer" containerID="0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144" Oct 30 21:00:18.779792 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:18.779755 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" path="/var/lib/kubelet/pods/3c94b98b-e7ee-44d9-8df3-86601cfa6f3f/volumes" Oct 30 21:00:18.885647 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:18.885559 5855 generic.go:356] "Generic (PLEG): container finished" podID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerID="6f95147f203d20edae6f4717331ed0f154ba2c0b925b55de9fe61c5e03a80a11" exitCode=60 Oct 30 21:00:18.885819 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:18.885638 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerDied","Data":"6f95147f203d20edae6f4717331ed0f154ba2c0b925b55de9fe61c5e03a80a11"} Oct 30 21:00:18.885819 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:18.885750 5855 scope.go:122] "RemoveContainer" containerID="0f9949c4dae3adb440c7b072f75519c2d5fb7a430f9b454d0a07769616320144" Oct 30 21:00:18.886211 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:18.886193 5855 scope.go:122] "RemoveContainer" containerID="6f95147f203d20edae6f4717331ed0f154ba2c0b925b55de9fe61c5e03a80a11" Oct 30 21:00:18.886496 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:18.886476 5855 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-hp74v_odh-ai-gateway-infra(78a811d8-1441-4211-8714-cc4a5eecfa3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" Oct 30 21:00:19.926744 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:19.926702 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:20.078459 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.078428 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:20.167352 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.167315 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\") pod \"78a811d8-1441-4211-8714-cc4a5eecfa3e\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " Oct 30 21:00:20.167549 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.167468 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\" (UniqueName: \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\") pod \"78a811d8-1441-4211-8714-cc4a5eecfa3e\" (UID: \"78a811d8-1441-4211-8714-cc4a5eecfa3e\") " Oct 30 21:00:20.167887 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.167857 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca" pod "78a811d8-1441-4211-8714-cc4a5eecfa3e" (UID: "78a811d8-1441-4211-8714-cc4a5eecfa3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:20.170240 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.170209 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542" pod "78a811d8-1441-4211-8714-cc4a5eecfa3e" (UID: "78a811d8-1441-4211-8714-cc4a5eecfa3e"). InnerVolumeSpecName "kube-api-access-hm542". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:20.268624 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.268587 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm542\" (UniqueName: \"kubernetes.io/projected/78a811d8-1441-4211-8714-cc4a5eecfa3e-kube-api-access-hm542\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.268624 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.268621 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a811d8-1441-4211-8714-cc4a5eecfa3e-maas-api-service-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.903146 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.903106 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" event={"ID":"78a811d8-1441-4211-8714-cc4a5eecfa3e","Type":"ContainerDied","Data":"00fe7842ccb3ed1540ddbb17a0a3eee6ce32228f49d5fd0dc1094286b3a2a659"} Oct 30 21:00:20.903146 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.903145 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v" Oct 30 21:00:20.903397 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.903183 5855 scope.go:122] "RemoveContainer" containerID="6f95147f203d20edae6f4717331ed0f154ba2c0b925b55de9fe61c5e03a80a11" Oct 30 21:00:20.922276 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.922237 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:20.926389 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:20.926363 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-hp74v"] Oct 30 21:00:21.023297 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:21.023259 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7"] Oct 30 21:00:21.025976 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:21.025947 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bf5d95c-ctdw7"] Oct 30 21:00:22.035059 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.035002 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr"] Oct 30 21:00:22.038080 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.038055 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782d3e08-nkhkr"] Oct 30 21:00:22.040489 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.040464 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm"] Oct 30 21:00:22.042557 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.042539 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e18f318-mggdm"] Oct 30 21:00:22.694201 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694168 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh"] Oct 30 21:00:22.694788 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694769 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694789 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694800 5855 cpu_manager.go:414] "containerMap: removing container" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694806 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694816 5855 cpu_manager.go:414] "containerMap: removing container" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694821 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694828 5855 cpu_manager.go:414] "containerMap: removing container" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" Oct 30 21:00:22.694845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694834 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694850 5855 cpu_manager.go:414] "containerMap: removing container" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694855 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694875 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694881 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694892 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694899 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694965 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694972 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15834ab-9d48-49b1-b6be-98655a5ec37c" containerName="istio-proxy" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694978 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694983 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694988 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694994 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="70309972-d5d4-48c2-9c39-8c76e633ad3c" containerName="istio-proxy" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.694999 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:22.695054 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.695004 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0bf314-5df2-425e-9bee-131a00148753" containerName="istio-proxy" Oct 30 21:00:22.699924 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.699894 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.708110 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.708082 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh"] Oct 30 21:00:22.708852 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.708828 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:00:22.709147 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.709129 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:00:22.709358 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.709341 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:00:22.709536 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.709521 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:22.780147 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.780113 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006919cd-370a-4fef-9ffa-ca267d4c54c4" path="/var/lib/kubelet/pods/006919cd-370a-4fef-9ffa-ca267d4c54c4/volumes" Oct 30 21:00:22.780555 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.780543 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" path="/var/lib/kubelet/pods/78a811d8-1441-4211-8714-cc4a5eecfa3e/volumes" Oct 30 21:00:22.780966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.780953 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859b27c6-f873-4228-9230-3277a05931da" path="/var/lib/kubelet/pods/859b27c6-f873-4228-9230-3277a05931da/volumes" Oct 30 21:00:22.781615 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.781601 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fa74c9-28cf-4a03-9e37-8f0894a43a8f" path="/var/lib/kubelet/pods/c0fa74c9-28cf-4a03-9e37-8f0894a43a8f/volumes" Oct 30 21:00:22.800816 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.800781 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.800967 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.800827 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.800967 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.800939 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl4fs\" (UniqueName: \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.902341 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.902296 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.902533 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.902368 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.902533 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.902440 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl4fs\" (UniqueName: \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.903235 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.903204 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.905120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.905083 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:22.915902 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:22.915858 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl4fs\" (UniqueName: \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\") pod \"maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:23.020842 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:23.020756 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:23.304388 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:00:23.304356 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec8ced7_201c_4401_8374_a709e6230b22.slice/crio-f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804 WatchSource:0}: Error finding container f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804: Status 404 returned error can't find the container with id f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804 Oct 30 21:00:23.304877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:23.304511 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh"] Oct 30 21:00:23.949114 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:23.949008 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" event={"ID":"2ec8ced7-201c-4401-8374-a709e6230b22","Type":"ContainerStarted","Data":"8ef6f6150f7861bb116e9f939081cf86c4040cc9e5a8d4879af03128127c1b78"} Oct 30 21:00:23.949277 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:23.949116 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" event={"ID":"2ec8ced7-201c-4401-8374-a709e6230b22","Type":"ContainerStarted","Data":"f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804"} Oct 30 21:00:23.966713 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:23.966599 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" podStartSLOduration=1.9665798730000001 podStartE2EDuration="1.966579873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:23.966037515 +0000 UTC m=+1237.324750262" watchObservedRunningTime="2026-10-30 21:00:23.966579873 +0000 UTC m=+1237.325292670" Oct 30 21:00:24.958244 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:24.958207 5855 generic.go:356] "Generic (PLEG): container finished" podID="2ec8ced7-201c-4401-8374-a709e6230b22" containerID="8ef6f6150f7861bb116e9f939081cf86c4040cc9e5a8d4879af03128127c1b78" exitCode=0 Oct 30 21:00:24.958643 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:24.958292 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" event={"ID":"2ec8ced7-201c-4401-8374-a709e6230b22","Type":"ContainerDied","Data":"8ef6f6150f7861bb116e9f939081cf86c4040cc9e5a8d4879af03128127c1b78"} Oct 30 21:00:26.034031 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.033987 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr"] Oct 30 21:00:26.037821 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.037781 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefa7b1c-xjjjr"] Oct 30 21:00:26.150707 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.150363 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:26.240263 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.240172 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\") pod \"2ec8ced7-201c-4401-8374-a709e6230b22\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " Oct 30 21:00:26.240444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.240293 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\") pod \"2ec8ced7-201c-4401-8374-a709e6230b22\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " Oct 30 21:00:26.240444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.240342 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\" (UniqueName: \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\") pod \"2ec8ced7-201c-4401-8374-a709e6230b22\" (UID: \"2ec8ced7-201c-4401-8374-a709e6230b22\") " Oct 30 21:00:26.240563 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.240513 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca" pod "2ec8ced7-201c-4401-8374-a709e6230b22" (UID: "2ec8ced7-201c-4401-8374-a709e6230b22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:26.240812 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.240793 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:26.242579 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.242551 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca" pod "2ec8ced7-201c-4401-8374-a709e6230b22" (UID: "2ec8ced7-201c-4401-8374-a709e6230b22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:26.242579 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.242575 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs" pod "2ec8ced7-201c-4401-8374-a709e6230b22" (UID: "2ec8ced7-201c-4401-8374-a709e6230b22"). InnerVolumeSpecName "kube-api-access-gl4fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:26.341950 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.341919 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec8ced7-201c-4401-8374-a709e6230b22-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:26.341950 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.341946 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl4fs\" (UniqueName: \"kubernetes.io/projected/2ec8ced7-201c-4401-8374-a709e6230b22-kube-api-access-gl4fs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:26.798685 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.798626 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179781f2-fc14-4b7e-b76c-5c8427e1ced7" path="/var/lib/kubelet/pods/179781f2-fc14-4b7e-b76c-5c8427e1ced7/volumes" Oct 30 21:00:26.974681 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.974627 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" Oct 30 21:00:26.974853 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.974690 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh" event={"ID":"2ec8ced7-201c-4401-8374-a709e6230b22","Type":"ContainerDied","Data":"f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804"} Oct 30 21:00:26.974853 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:26.974738 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f735f7b32a837a09fb6a8d6018a9d755783f4198fcc3d25f9fb0e12acdfa0804" Oct 30 21:00:27.560697 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:27.560641 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 21:00:28.024594 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:28.024506 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6"] Oct 30 21:00:28.026941 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:28.026914 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd3b57-5fch6"] Oct 30 21:00:28.778986 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:28.778950 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820b0b26-5bdd-435c-abb1-cf7af82b181c" path="/var/lib/kubelet/pods/820b0b26-5bdd-435c-abb1-cf7af82b181c/volumes" Oct 30 21:00:29.022698 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:29.022615 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt"] Oct 30 21:00:29.025762 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:29.025732 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cee663-xkmwt"] Oct 30 21:00:30.777829 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:30.777799 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b953826-67de-4def-827f-23e6d6e3d414" path="/var/lib/kubelet/pods/4b953826-67de-4def-827f-23e6d6e3d414/volumes" Oct 30 21:00:30.925641 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:30.925598 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:00:32.535164 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.535130 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 21:00:32.536108 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.535355 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" containerID="cri-o://043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7" gracePeriod=30 Oct 30 21:00:32.538459 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.538432 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 21:00:32.538688 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.538646 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" containerID="cri-o://f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0" gracePeriod=30 Oct 30 21:00:32.541008 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.540981 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 21:00:32.541239 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:32.541187 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" containerID="cri-o://b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1" gracePeriod=30 Oct 30 21:00:33.082737 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:33.082705 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn"] Oct 30 21:00:33.085744 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:33.085708 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a067a0da-kzccn"] Oct 30 21:00:34.045274 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045228 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z"] Oct 30 21:00:34.045925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045905 5855 cpu_manager.go:414] "containerMap: removing container" podUID="2ec8ced7-201c-4401-8374-a709e6230b22" containerName="revoke-keys" Oct 30 21:00:34.046005 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045925 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec8ced7-201c-4401-8374-a709e6230b22" containerName="revoke-keys" Oct 30 21:00:34.046005 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045952 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:34.046005 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045960 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c94b98b-e7ee-44d9-8df3-86601cfa6f3f" containerName="cleanup" Oct 30 21:00:34.046005 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045977 5855 cpu_manager.go:414] "containerMap: removing container" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:34.046005 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.045985 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:34.046251 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.046103 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec8ced7-201c-4401-8374-a709e6230b22" containerName="revoke-keys" Oct 30 21:00:34.046251 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.046118 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a811d8-1441-4211-8714-cc4a5eecfa3e" containerName="cleanup" Oct 30 21:00:34.056128 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.056093 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.060591 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.060549 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:34.060808 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.060598 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:00:34.061728 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.061706 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:00:34.061834 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.061770 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:00:34.070525 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.070477 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z"] Oct 30 21:00:34.147249 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.147208 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.147443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.147280 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzstv\" (UniqueName: \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.147443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.147393 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.248785 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.248740 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.248981 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.248812 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzstv\" (UniqueName: \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.248981 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.248854 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.249586 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.249562 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.252013 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.251977 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.262638 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.262604 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzstv\" (UniqueName: \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\") pod \"maas-api-revoke-sub-bf7adbb7-6ng2z\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.373839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.373746 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:34.740601 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.740576 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z"] Oct 30 21:00:34.742994 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:00:34.742970 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fceb8e_1b0e_4aa3_81d7_236f42183ee2.slice/crio-4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9 WatchSource:0}: Error finding container 4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9: Status 404 returned error can't find the container with id 4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9 Oct 30 21:00:34.779321 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:34.779290 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f22af8e-6941-4033-af47-157af709f986" path="/var/lib/kubelet/pods/2f22af8e-6941-4033-af47-157af709f986/volumes" Oct 30 21:00:35.059013 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:35.058933 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p"] Oct 30 21:00:35.060477 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:35.060442 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac26579-hq62p"] Oct 30 21:00:35.100538 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:35.100501 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" event={"ID":"52fceb8e-1b0e-4aa3-81d7-236f42183ee2","Type":"ContainerStarted","Data":"2a7e776be0a8a372ab605ee18ebf31205ac1629351194780e1777d164f4b7495"} Oct 30 21:00:35.100786 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:35.100559 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" event={"ID":"52fceb8e-1b0e-4aa3-81d7-236f42183ee2","Type":"ContainerStarted","Data":"4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9"} Oct 30 21:00:35.126120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:35.126067 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" podStartSLOduration=1.126050569 podStartE2EDuration="1.126050569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:35.123939504 +0000 UTC m=+1248.482652250" watchObservedRunningTime="2026-10-30 21:00:35.126050569 +0000 UTC m=+1248.484763325" Oct 30 21:00:36.111208 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:36.111172 5855 generic.go:356] "Generic (PLEG): container finished" podID="52fceb8e-1b0e-4aa3-81d7-236f42183ee2" containerID="2a7e776be0a8a372ab605ee18ebf31205ac1629351194780e1777d164f4b7495" exitCode=0 Oct 30 21:00:36.111777 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:36.111259 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" event={"ID":"52fceb8e-1b0e-4aa3-81d7-236f42183ee2","Type":"ContainerDied","Data":"2a7e776be0a8a372ab605ee18ebf31205ac1629351194780e1777d164f4b7495"} Oct 30 21:00:36.808748 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:36.808372 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5278b0-2192-40b2-b320-a87816377803" path="/var/lib/kubelet/pods/bb5278b0-2192-40b2-b320-a87816377803/volumes" Oct 30 21:00:37.259773 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.259735 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:37.392452 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.392349 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\") pod \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " Oct 30 21:00:37.392643 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.392485 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\") pod \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " Oct 30 21:00:37.392643 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.392534 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\" (UniqueName: \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\") pod \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\" (UID: \"52fceb8e-1b0e-4aa3-81d7-236f42183ee2\") " Oct 30 21:00:37.392815 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.392769 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca" pod "52fceb8e-1b0e-4aa3-81d7-236f42183ee2" (UID: "52fceb8e-1b0e-4aa3-81d7-236f42183ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:37.393049 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.393026 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.394877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.394843 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca" pod "52fceb8e-1b0e-4aa3-81d7-236f42183ee2" (UID: "52fceb8e-1b0e-4aa3-81d7-236f42183ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:37.395004 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.394848 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv" pod "52fceb8e-1b0e-4aa3-81d7-236f42183ee2" (UID: "52fceb8e-1b0e-4aa3-81d7-236f42183ee2"). InnerVolumeSpecName "kube-api-access-bzstv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.494264 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.494230 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.494264 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.494256 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzstv\" (UniqueName: \"kubernetes.io/projected/52fceb8e-1b0e-4aa3-81d7-236f42183ee2-kube-api-access-bzstv\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.874465 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.874430 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 21:00:37.897548 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897507 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897597 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897641 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897752 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897984 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897816 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897984 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897846 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.897984 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897881 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897984 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897911 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.897984 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.897935 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.898249 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898010 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.898249 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898104 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\") pod \"30f8ac57-eaae-4e5d-93d2-499338eddf24\" (UID: \"30f8ac57-eaae-4e5d-93d2-499338eddf24\") " Oct 30 21:00:37.898249 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898161 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.898410 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898287 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:37.898578 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898552 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.898578 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898580 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.898937 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898594 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.898937 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898607 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f8ac57-eaae-4e5d-93d2-499338eddf24-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.898937 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.898878 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.901700 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.901592 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:37.901700 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.901600 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:37.903254 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.903221 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.904666 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.904578 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt" pod "30f8ac57-eaae-4e5d-93d2-499338eddf24" (UID: "30f8ac57-eaae-4e5d-93d2-499338eddf24"). InnerVolumeSpecName "kube-api-access-fvxjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.932941 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.932904 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 21:00:37.965795 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:37.965761 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001768 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001822 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001875 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001921 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001952 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.001987 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002033 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002109 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002105 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002149 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002190 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002227 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002267 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002298 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002334 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.002356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002373 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002418 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002485 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\") pod \"99c5f684-595f-40e1-8549-c8a3b48e2866\" (UID: \"99c5f684-595f-40e1-8549-c8a3b48e2866\") " Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002503 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.002517 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\") pod \"076643e8-713c-4c2f-acd3-651855e837c3\" (UID: \"076643e8-713c-4c2f-acd3-651855e837c3\") " Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003040 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003065 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003069 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003081 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003119 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003137 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvxjt\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-kube-api-access-fvxjt\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003155 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003168 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f8ac57-eaae-4e5d-93d2-499338eddf24-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.003738 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.003723 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.004463 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.004015 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:38.007790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.006851 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.007790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.007010 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.007790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.007212 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.007790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.007440 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:38.008354 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.007870 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "kube-api-access-6zmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:38.008786 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.008761 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.009669 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.009615 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.011804 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.011112 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:38.011804 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.011579 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:38.012168 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.012142 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:38.014122 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.014098 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:38.014238 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.014193 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz" pod "99c5f684-595f-40e1-8549-c8a3b48e2866" (UID: "99c5f684-595f-40e1-8549-c8a3b48e2866"). InnerVolumeSpecName "kube-api-access-h84zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:38.014715 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.014646 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token" pod "076643e8-713c-4c2f-acd3-651855e837c3" (UID: "076643e8-713c-4c2f-acd3-651855e837c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:38.104391 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104356 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104391 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104387 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104391 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104397 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104407 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104415 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104424 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84zz\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-kube-api-access-h84zz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104434 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076643e8-713c-4c2f-acd3-651855e837c3-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104443 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104451 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104459 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c5f684-595f-40e1-8549-c8a3b48e2866-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104466 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104477 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104486 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zmq9\" (UniqueName: \"kubernetes.io/projected/076643e8-713c-4c2f-acd3-651855e837c3-kube-api-access-6zmq9\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104494 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076643e8-713c-4c2f-acd3-651855e837c3-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104502 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076643e8-713c-4c2f-acd3-651855e837c3-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.104632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.104511 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c5f684-595f-40e1-8549-c8a3b48e2866-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:38.129885 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.129844 5855 generic.go:356] "Generic (PLEG): container finished" podID="076643e8-713c-4c2f-acd3-651855e837c3" containerID="f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0" exitCode=0 Oct 30 21:00:38.130089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.129966 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" Oct 30 21:00:38.130089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.129970 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" event={"ID":"076643e8-713c-4c2f-acd3-651855e837c3","Type":"ContainerDied","Data":"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0"} Oct 30 21:00:38.130227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.130100 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4" event={"ID":"076643e8-713c-4c2f-acd3-651855e837c3","Type":"ContainerDied","Data":"8acd3ea2439b2bf6cdb26531a2b34d0ea9c5b52f8a857a1dddcefaeee00ac444"} Oct 30 21:00:38.130227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.130127 5855 scope.go:122] "RemoveContainer" containerID="f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0" Oct 30 21:00:38.132764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.132580 5855 generic.go:356] "Generic (PLEG): container finished" podID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerID="043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7" exitCode=0 Oct 30 21:00:38.132764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.132623 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" event={"ID":"30f8ac57-eaae-4e5d-93d2-499338eddf24","Type":"ContainerDied","Data":"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7"} Oct 30 21:00:38.132764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.132706 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" Oct 30 21:00:38.132764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.132720 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6" event={"ID":"30f8ac57-eaae-4e5d-93d2-499338eddf24","Type":"ContainerDied","Data":"a8510da3c6dea9a6a6f47be6718527c3f49bad65f22ce215b15292f471ce1564"} Oct 30 21:00:38.135440 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.135405 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" event={"ID":"52fceb8e-1b0e-4aa3-81d7-236f42183ee2","Type":"ContainerDied","Data":"4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9"} Oct 30 21:00:38.135617 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.135467 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c54b439a06e5dc1e663f50d2ac2eff2e6b43bda205297000f899d1a49347bd9" Oct 30 21:00:38.135617 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.135556 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z" Oct 30 21:00:38.137965 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.137937 5855 generic.go:356] "Generic (PLEG): container finished" podID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerID="b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1" exitCode=0 Oct 30 21:00:38.138116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.137976 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" Oct 30 21:00:38.138116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.138034 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" event={"ID":"99c5f684-595f-40e1-8549-c8a3b48e2866","Type":"ContainerDied","Data":"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1"} Oct 30 21:00:38.138116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.138075 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd" event={"ID":"99c5f684-595f-40e1-8549-c8a3b48e2866","Type":"ContainerDied","Data":"d2b7389e9a49d5a8a43c3ac4c277e70b7bd9bb8bd08da0819979546d09804eb2"} Oct 30 21:00:38.144426 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.144405 5855 scope.go:122] "RemoveContainer" containerID="f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0" Oct 30 21:00:38.144787 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:38.144761 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0\": container with ID starting with f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0 not found: ID does not exist" containerID="f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0" Oct 30 21:00:38.144905 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.144812 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0"} err="failed to get container status \"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0\": rpc error: code = NotFound desc = could not find container \"f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0\": container with ID starting with f36b0384d03ac963ae3d08ca1da717fdc9b8d8b427cf707232c8e883c517edb0 not found: ID does not exist" Oct 30 21:00:38.144905 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.144830 5855 scope.go:122] "RemoveContainer" containerID="043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7" Oct 30 21:00:38.156475 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.156453 5855 scope.go:122] "RemoveContainer" containerID="043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7" Oct 30 21:00:38.156880 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:38.156855 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7\": container with ID starting with 043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7 not found: ID does not exist" containerID="043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7" Oct 30 21:00:38.156966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.156891 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7"} err="failed to get container status \"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7\": rpc error: code = NotFound desc = could not find container \"043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7\": container with ID starting with 043ca2ab0a3b266dfd7f45e7acb6b99dc6ba3ad52d48f22f11a5cd486f8f2eb7 not found: ID does not exist" Oct 30 21:00:38.156966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.156913 5855 scope.go:122] "RemoveContainer" containerID="b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1" Oct 30 21:00:38.170008 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.169982 5855 scope.go:122] "RemoveContainer" containerID="b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1" Oct 30 21:00:38.170421 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:38.170394 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1\": container with ID starting with b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1 not found: ID does not exist" containerID="b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1" Oct 30 21:00:38.170553 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.170429 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1"} err="failed to get container status \"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1\": rpc error: code = NotFound desc = could not find container \"b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1\": container with ID starting with b249a4dc6b90a4784229a3f3c492f64e1211b013383782bcd13b8b2693c8a9b1 not found: ID does not exist" Oct 30 21:00:38.176503 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.176472 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 21:00:38.188920 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.188880 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-h4pz4"] Oct 30 21:00:38.223322 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.223284 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 21:00:38.229714 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.229681 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-2qdm6"] Oct 30 21:00:38.251279 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.251251 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 21:00:38.263665 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.263614 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qn9gd"] Oct 30 21:00:38.779011 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.778980 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076643e8-713c-4c2f-acd3-651855e837c3" path="/var/lib/kubelet/pods/076643e8-713c-4c2f-acd3-651855e837c3/volumes" Oct 30 21:00:38.779461 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.779448 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" path="/var/lib/kubelet/pods/30f8ac57-eaae-4e5d-93d2-499338eddf24/volumes" Oct 30 21:00:38.779888 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:38.779874 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" path="/var/lib/kubelet/pods/99c5f684-595f-40e1-8549-c8a3b48e2866/volumes" Oct 30 21:00:42.582849 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.582797 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" containerID="cri-o://6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a" gracePeriod=45 Oct 30 21:00:42.874768 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.874745 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 21:00:42.969052 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969022 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.969217 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969120 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.969217 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969185 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.969217 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969215 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.969409 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969260 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\" (UniqueName: \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.969409 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.969322 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\") pod \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\" (UID: \"690b2c59-3c3e-43b1-88b1-29aaff3a00d8\") " Oct 30 21:00:42.970228 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.970197 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:42.970422 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.970384 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:42.970570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.970451 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:42.972485 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.972444 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:42.973090 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.973067 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:42.973192 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:42.973117 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69" pod "690b2c59-3c3e-43b1-88b1-29aaff3a00d8" (UID: "690b2c59-3c3e-43b1-88b1-29aaff3a00d8"). InnerVolumeSpecName "kube-api-access-26k69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071195 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071225 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071239 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071251 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071267 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26k69\" (UniqueName: \"kubernetes.io/projected/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-kube-api-access-26k69\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.073963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.071281 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/690b2c59-3c3e-43b1-88b1-29aaff3a00d8-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.180899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.180792 5855 generic.go:356] "Generic (PLEG): container finished" podID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerID="6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a" exitCode=0 Oct 30 21:00:43.180899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.180851 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" Oct 30 21:00:43.180899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.180874 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" event={"ID":"690b2c59-3c3e-43b1-88b1-29aaff3a00d8","Type":"ContainerDied","Data":"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a"} Oct 30 21:00:43.181195 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.180921 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv" event={"ID":"690b2c59-3c3e-43b1-88b1-29aaff3a00d8","Type":"ContainerDied","Data":"c87adf928fb23f1cf137816789fa4228567b19141d7fdf3e006cade8689e7321"} Oct 30 21:00:43.181195 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.180940 5855 scope.go:122] "RemoveContainer" containerID="6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a" Oct 30 21:00:43.192100 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.192078 5855 scope.go:122] "RemoveContainer" containerID="6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a" Oct 30 21:00:43.192498 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:00:43.192463 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a\": container with ID starting with 6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a not found: ID does not exist" containerID="6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a" Oct 30 21:00:43.192628 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.192506 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a"} err="failed to get container status \"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a\": rpc error: code = NotFound desc = could not find container \"6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a\": container with ID starting with 6adeecbea51c586f45a97347a998bb5c005d7596ae063ae25ed574985332e94a not found: ID does not exist" Oct 30 21:00:43.205074 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.205035 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 21:00:43.213556 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:43.213522 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/simulator-w1-b8aee1-kserve-67fc4dddcd-952mv"] Oct 30 21:00:44.023758 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:44.023725 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n"] Oct 30 21:00:44.026495 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:44.026465 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edba010c-gpw2n"] Oct 30 21:00:44.779561 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:44.779516 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" path="/var/lib/kubelet/pods/690b2c59-3c3e-43b1-88b1-29aaff3a00d8/volumes" Oct 30 21:00:44.780145 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:44.780127 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d534605-ae0b-48c6-a461-7db789d16547" path="/var/lib/kubelet/pods/7d534605-ae0b-48c6-a461-7db789d16547/volumes" Oct 30 21:00:47.960870 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:47.960835 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:00:48.458155 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:48.458123 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh"] Oct 30 21:00:48.467312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:48.467281 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b8aee1-6nklh"] Oct 30 21:00:48.786881 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:48.786702 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec8ced7-201c-4401-8374-a709e6230b22" path="/var/lib/kubelet/pods/2ec8ced7-201c-4401-8374-a709e6230b22/volumes" Oct 30 21:00:49.683683 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.679294 5855 scope.go:122] "RemoveContainer" containerID="78b68fe916f6e63c34b2c3b4bb4210e293fd65ed69fc56011558ec6c084aa8d2" Oct 30 21:00:49.696627 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.696467 5855 scope.go:122] "RemoveContainer" containerID="b1aeda06b3e534dbc0c7cbacbffa7f88c4000796df2176e22d0ab2bf8133ec51" Oct 30 21:00:49.714998 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.714973 5855 scope.go:122] "RemoveContainer" containerID="ff03e21397a2ad72ba984757f1451c859e468b9d25b7cac2ae673e1be2c90b92" Oct 30 21:00:49.727586 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.727568 5855 scope.go:122] "RemoveContainer" containerID="42df6366cdd4abcdfef1deface41cd21bf104bb257af3e780d1478095bdced32" Oct 30 21:00:49.742257 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.742231 5855 scope.go:122] "RemoveContainer" containerID="ec2f445e1df274eaae1f9b164869d92703141728f8fcb005e4f45a0937984297" Oct 30 21:00:49.764069 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.764040 5855 scope.go:122] "RemoveContainer" containerID="9c65c2d2d076d9e8fe716cd676607d36b14b893e8c926b72589a602d43a2fb2d" Oct 30 21:00:49.777255 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.777231 5855 scope.go:122] "RemoveContainer" containerID="1337268f2bb71982d97571b4fdd5127f13c983cf7d4a99421166e61bf46760b0" Oct 30 21:00:49.787495 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.787472 5855 scope.go:122] "RemoveContainer" containerID="fdb0fed42bb0848bc61c09264cd4fa8eb91f7dd76c0634145241e13e5bed1a72" Oct 30 21:00:49.799060 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.799034 5855 scope.go:122] "RemoveContainer" containerID="e911993e35734c489eb298a900a3a10b5510590f50efa108179dd84e84decee3" Oct 30 21:00:49.811702 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.811672 5855 scope.go:122] "RemoveContainer" containerID="b39a3e171fbacfdea14f07a11e4b7899b702bbc101266b87fdcffadf15190833" Oct 30 21:00:49.822317 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:49.822283 5855 scope.go:122] "RemoveContainer" containerID="6abff57efdf5a5030128f690e24779c45156d9d963a9bf6cd5c221729c7ea8b7" Oct 30 21:00:50.033594 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:50.033550 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl"] Oct 30 21:00:50.036927 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:50.036896 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1f5014-g9cxl"] Oct 30 21:00:50.778086 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:50.778052 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05177ebd-27f4-479a-b459-816cb238dc4b" path="/var/lib/kubelet/pods/05177ebd-27f4-479a-b459-816cb238dc4b/volumes" Oct 30 21:00:57.286785 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.286685 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 21:00:57.287278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.286942 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" containerID="cri-o://34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79" gracePeriod=30 Oct 30 21:00:57.289815 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.289784 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 21:00:57.290071 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.290027 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" containerID="cri-o://871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a" gracePeriod=30 Oct 30 21:00:57.303963 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.303919 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 21:00:57.304188 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.304161 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" containerID="cri-o://1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61" gracePeriod=30 Oct 30 21:00:57.986161 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:00:57.986119 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:01:00.925290 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:01:00.925241 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:01:02.622935 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.622762 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 21:01:02.634160 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.634132 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 21:01:02.673056 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.673023 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 21:01:02.756133 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756104 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.756312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756144 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.756312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756176 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.756447 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756395 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.756536 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756505 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.756763 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756694 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.756763 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756706 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756764 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756806 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756878 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756892 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756919 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.756956 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.756946 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757006 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757039 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757067 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757099 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757135 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757170 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757183 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757201 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757227 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757263 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757292 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757341 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757370 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757405 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757460 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757544 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\") pod \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\" (UID: \"22f57bb2-f46f-42f0-8b48-5712dcb3eae2\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757587 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\") pod \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\" (UID: \"18e90219-2c2a-4393-b7a1-dbbd57fc0e32\") " Oct 30 21:01:02.757799 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.757617 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\") pod \"3f057ef4-4650-4566-bf29-0df595a4c5f2\" (UID: \"3f057ef4-4650-4566-bf29-0df595a4c5f2\") " Oct 30 21:01:02.758868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.758627 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.758868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.758677 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.758868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.758693 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f057ef4-4650-4566-bf29-0df595a4c5f2-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.758868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.758707 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.759356 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759301 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.759442 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759394 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.759728 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759541 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.760114 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759857 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.760114 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759934 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.760114 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.759961 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.760546 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.760487 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.760546 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.760557 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.760823 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.760723 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.760893 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.760820 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.761371 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.761004 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.761371 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.761019 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.761371 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.761291 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.761371 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.761304 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.761719 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.761696 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.762531 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.762507 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.762846 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.762739 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "kube-api-access-dvrzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.762846 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.762769 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.764159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.763898 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "kube-api-access-gnhlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.764159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.763993 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy" pod "3f057ef4-4650-4566-bf29-0df595a4c5f2" (UID: "3f057ef4-4650-4566-bf29-0df595a4c5f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.764159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.764072 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "kube-api-access-bmz8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.764476 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.764459 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token" pod "22f57bb2-f46f-42f0-8b48-5712dcb3eae2" (UID: "22f57bb2-f46f-42f0-8b48-5712dcb3eae2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.764545 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.764524 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo" pod "18e90219-2c2a-4393-b7a1-dbbd57fc0e32" (UID: "18e90219-2c2a-4393-b7a1-dbbd57fc0e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860087 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860122 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860137 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhlq\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-kube-api-access-gnhlq\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860150 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmz8f\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-kube-api-access-bmz8f\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860163 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860175 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860184 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860187 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860201 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860212 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860223 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860234 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f057ef4-4650-4566-bf29-0df595a4c5f2-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860246 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860257 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrzw\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-kube-api-access-dvrzw\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860269 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860280 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860292 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860303 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860316 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860327 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860339 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e90219-2c2a-4393-b7a1-dbbd57fc0e32-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860351 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860363 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f57bb2-f46f-42f0-8b48-5712dcb3eae2-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.860570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:02.860376 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f057ef4-4650-4566-bf29-0df595a4c5f2-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:03.382561 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.382521 5855 generic.go:356] "Generic (PLEG): container finished" podID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerID="871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a" exitCode=0 Oct 30 21:01:03.382778 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.382599 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" Oct 30 21:01:03.382778 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.382607 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" event={"ID":"22f57bb2-f46f-42f0-8b48-5712dcb3eae2","Type":"ContainerDied","Data":"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a"} Oct 30 21:01:03.382778 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.382710 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8" event={"ID":"22f57bb2-f46f-42f0-8b48-5712dcb3eae2","Type":"ContainerDied","Data":"ba7c003f74fdd662425aeb4104bfad1ef4be9d20dd5ee9a3471c938d4ccd0fdd"} Oct 30 21:01:03.382778 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.382743 5855 scope.go:122] "RemoveContainer" containerID="871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a" Oct 30 21:01:03.384570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.384529 5855 generic.go:356] "Generic (PLEG): container finished" podID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerID="1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61" exitCode=0 Oct 30 21:01:03.384699 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.384598 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" Oct 30 21:01:03.384775 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.384759 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" event={"ID":"18e90219-2c2a-4393-b7a1-dbbd57fc0e32","Type":"ContainerDied","Data":"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61"} Oct 30 21:01:03.385514 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.384812 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn" event={"ID":"18e90219-2c2a-4393-b7a1-dbbd57fc0e32","Type":"ContainerDied","Data":"ad799c8c9fa2ed03869e0a9321ff05ca07b391e96a2ab0cd69d699fb2b0b8bea"} Oct 30 21:01:03.388813 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.388779 5855 generic.go:356] "Generic (PLEG): container finished" podID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerID="34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79" exitCode=0 Oct 30 21:01:03.388971 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.388848 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" event={"ID":"3f057ef4-4650-4566-bf29-0df595a4c5f2","Type":"ContainerDied","Data":"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79"} Oct 30 21:01:03.388971 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.388868 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" Oct 30 21:01:03.388971 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.388898 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg" event={"ID":"3f057ef4-4650-4566-bf29-0df595a4c5f2","Type":"ContainerDied","Data":"ab438e713866252413f81703a6a905ec900889d9534316c61bcbf6602380e34c"} Oct 30 21:01:03.395983 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.395961 5855 scope.go:122] "RemoveContainer" containerID="871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a" Oct 30 21:01:03.396342 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:01:03.396323 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a\": container with ID starting with 871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a not found: ID does not exist" containerID="871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a" Oct 30 21:01:03.396393 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.396351 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a"} err="failed to get container status \"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a\": rpc error: code = NotFound desc = could not find container \"871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a\": container with ID starting with 871b145ab087e80b66d3d92f042230e69e187fb053f14884e3df7fd4ef4fcd6a not found: ID does not exist" Oct 30 21:01:03.396393 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.396370 5855 scope.go:122] "RemoveContainer" containerID="1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61" Oct 30 21:01:03.407932 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.407906 5855 scope.go:122] "RemoveContainer" containerID="1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61" Oct 30 21:01:03.408228 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:01:03.408207 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61\": container with ID starting with 1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61 not found: ID does not exist" containerID="1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61" Oct 30 21:01:03.408310 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.408240 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61"} err="failed to get container status \"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61\": rpc error: code = NotFound desc = could not find container \"1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61\": container with ID starting with 1305090c7157495a45d007a8908d6e939c8995af74d1e2804b0df50545912e61 not found: ID does not exist" Oct 30 21:01:03.408310 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.408257 5855 scope.go:122] "RemoveContainer" containerID="34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79" Oct 30 21:01:03.414269 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.414245 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 21:01:03.417883 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.417858 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-fwzkn"] Oct 30 21:01:03.420580 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.420462 5855 scope.go:122] "RemoveContainer" containerID="34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79" Oct 30 21:01:03.420825 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:01:03.420807 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79\": container with ID starting with 34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79 not found: ID does not exist" containerID="34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79" Oct 30 21:01:03.420876 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.420834 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79"} err="failed to get container status \"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79\": rpc error: code = NotFound desc = could not find container \"34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79\": container with ID starting with 34eb6e789269ea4cb8ef327d1f8221493c57906201cce3f388291c2ec3dc7f79 not found: ID does not exist" Oct 30 21:01:03.434056 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.434024 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 21:01:03.441116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.441083 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-cp7lg"] Oct 30 21:01:03.453827 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.453788 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 21:01:03.458865 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:03.458833 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-q52g8"] Oct 30 21:01:04.778236 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:04.778189 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" path="/var/lib/kubelet/pods/18e90219-2c2a-4393-b7a1-dbbd57fc0e32/volumes" Oct 30 21:01:04.778887 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:04.778868 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" path="/var/lib/kubelet/pods/22f57bb2-f46f-42f0-8b48-5712dcb3eae2/volumes" Oct 30 21:01:04.779362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:04.779343 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" path="/var/lib/kubelet/pods/3f057ef4-4650-4566-bf29-0df595a4c5f2/volumes" Oct 30 21:01:27.023912 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:27.023874 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr"] Oct 30 21:01:27.025247 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:27.025227 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06fc8a20-xnxfr"] Oct 30 21:01:28.778436 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:28.778401 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dff7848-04cc-4a2a-96e5-a5e64af18c30" path="/var/lib/kubelet/pods/6dff7848-04cc-4a2a-96e5-a5e64af18c30/volumes" Oct 30 21:01:30.924711 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:01:30.924673 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:01:37.773794 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.773762 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq"] Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774158 5855 cpu_manager.go:414] "containerMap: removing container" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774169 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774181 5855 cpu_manager.go:414] "containerMap: removing container" podUID="52fceb8e-1b0e-4aa3-81d7-236f42183ee2" containerName="revoke-keys" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774186 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fceb8e-1b0e-4aa3-81d7-236f42183ee2" containerName="revoke-keys" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774193 5855 cpu_manager.go:414] "containerMap: removing container" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774198 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774205 5855 cpu_manager.go:414] "containerMap: removing container" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774209 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774218 5855 cpu_manager.go:414] "containerMap: removing container" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774222 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774228 5855 cpu_manager.go:414] "containerMap: removing container" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774233 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774239 5855 cpu_manager.go:414] "containerMap: removing container" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774244 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774253 5855 cpu_manager.go:414] "containerMap: removing container" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774258 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774308 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c5f684-595f-40e1-8549-c8a3b48e2866" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774316 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fceb8e-1b0e-4aa3-81d7-236f42183ee2" containerName="revoke-keys" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774321 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f57bb2-f46f-42f0-8b48-5712dcb3eae2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774326 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="690b2c59-3c3e-43b1-88b1-29aaff3a00d8" containerName="main" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774333 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="076643e8-713c-4c2f-acd3-651855e837c3" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774338 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f057ef4-4650-4566-bf29-0df595a4c5f2" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774342 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e90219-2c2a-4393-b7a1-dbbd57fc0e32" containerName="istio-proxy" Oct 30 21:01:37.774414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.774349 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f8ac57-eaae-4e5d-93d2-499338eddf24" containerName="istio-proxy" Oct 30 21:01:37.777130 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.777108 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.787593 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.786477 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:01:37.787593 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.786996 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:01:37.790220 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.790185 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq"] Oct 30 21:01:37.852502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.852463 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:01:37.852502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.852463 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:01:37.874051 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.874013 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.874051 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.874055 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rcc\" (UniqueName: \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.874275 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.874151 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.975648 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.975612 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.975893 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.975705 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rcc\" (UniqueName: \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.975893 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.975763 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.976414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.976389 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.978249 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.978231 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:37.988927 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:37.988901 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rcc\" (UniqueName: \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\") pod \"maas-api-revoke-keys-e2e-mt-812bfe65-47qvq\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:38.095124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:38.095091 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:38.370874 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:38.370837 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq"] Oct 30 21:01:38.372178 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:01:38.372149 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63f9a62_dfb9_4a32_9d55_3fd851792cfe.slice/crio-65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77 WatchSource:0}: Error finding container 65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77: Status 404 returned error can't find the container with id 65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77 Oct 30 21:01:38.645223 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:38.645128 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" event={"ID":"c63f9a62-dfb9-4a32-9d55-3fd851792cfe","Type":"ContainerStarted","Data":"65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77"} Oct 30 21:01:39.652919 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:39.652881 5855 generic.go:356] "Generic (PLEG): container finished" podID="c63f9a62-dfb9-4a32-9d55-3fd851792cfe" containerID="5b162f37409fb8765663b6bef097167afb0465333392e926808f9b707a7e2de9" exitCode=0 Oct 30 21:01:39.653330 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:39.652962 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" event={"ID":"c63f9a62-dfb9-4a32-9d55-3fd851792cfe","Type":"ContainerDied","Data":"5b162f37409fb8765663b6bef097167afb0465333392e926808f9b707a7e2de9"} Oct 30 21:01:40.807520 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.807489 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:40.919448 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.919340 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\" (UniqueName: \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\") pod \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " Oct 30 21:01:40.919448 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.919412 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\") pod \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " Oct 30 21:01:40.919780 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.919451 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\") pod \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\" (UID: \"c63f9a62-dfb9-4a32-9d55-3fd851792cfe\") " Oct 30 21:01:40.919913 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.919885 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca" pod "c63f9a62-dfb9-4a32-9d55-3fd851792cfe" (UID: "c63f9a62-dfb9-4a32-9d55-3fd851792cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:40.921855 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.921827 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca" pod "c63f9a62-dfb9-4a32-9d55-3fd851792cfe" (UID: "c63f9a62-dfb9-4a32-9d55-3fd851792cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:40.921969 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:40.921870 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc" pod "c63f9a62-dfb9-4a32-9d55-3fd851792cfe" (UID: "c63f9a62-dfb9-4a32-9d55-3fd851792cfe"). InnerVolumeSpecName "kube-api-access-j9rcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:41.024691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.022455 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rcc\" (UniqueName: \"kubernetes.io/projected/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-kube-api-access-j9rcc\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:41.024691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.022502 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:41.024691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.022517 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63f9a62-dfb9-4a32-9d55-3fd851792cfe-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:41.667882 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.667851 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" Oct 30 21:01:41.668075 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.667890 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq" event={"ID":"c63f9a62-dfb9-4a32-9d55-3fd851792cfe","Type":"ContainerDied","Data":"65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77"} Oct 30 21:01:41.668075 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:41.667951 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65eb9625120a1147adb5ac2ea3eec519165a867809b52b6a975c4b44bd5cfb77" Oct 30 21:01:48.442279 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:48.442242 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq"] Oct 30 21:01:48.444561 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:48.444537 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-812bfe65-47qvq"] Oct 30 21:01:48.778322 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:48.778241 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63f9a62-dfb9-4a32-9d55-3fd851792cfe" path="/var/lib/kubelet/pods/c63f9a62-dfb9-4a32-9d55-3fd851792cfe/volumes" Oct 30 21:01:49.959690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:49.959643 5855 scope.go:122] "RemoveContainer" containerID="5d95b19a84065f5a99a96ae15e86ab39cdb3b65659b84772e4d3e3f2372d2096" Oct 30 21:01:49.969358 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:49.969335 5855 scope.go:122] "RemoveContainer" containerID="edd0a0337dbc741ce7033964782d03709b951e2a80a7e0d3a20ef540252ced65" Oct 30 21:01:49.979445 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:49.979408 5855 scope.go:122] "RemoveContainer" containerID="2eef43a18d601ee6b984b7e69b73a8f2e0ff94e3b26b702a5283383cbd5e66f8" Oct 30 21:01:49.991679 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:49.991638 5855 scope.go:122] "RemoveContainer" containerID="af7725722fbdd930cd4d60eb1d8f60d6cbe46673069048f2faff8de45a1f5854" Oct 30 21:01:50.044572 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.044532 5855 scope.go:122] "RemoveContainer" containerID="c1ea1d4b436f89ac0976fedcf355cc497092a4cee6bd11e9b78a1364d44f0742" Oct 30 21:01:50.055193 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.055171 5855 scope.go:122] "RemoveContainer" containerID="f7cdff769eee65506a9c5f7aa41d60a15365ded954d3ee08c708630f1f5b7f91" Oct 30 21:01:50.065137 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.065117 5855 scope.go:122] "RemoveContainer" containerID="920890d07b8fe504ff0bb5a4181065f83c8d6ea428189870cb73f02e0dea374d" Oct 30 21:01:50.074828 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.074805 5855 scope.go:122] "RemoveContainer" containerID="196e8d08230bc6a83f48872052028ae4271ac65aef75419f84465a7ac952e704" Oct 30 21:01:50.084506 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.084474 5855 scope.go:122] "RemoveContainer" containerID="246657fb1142b9ffb548cb4cf58ef48612665979a34cfa267d25dcb1c08bfea2" Oct 30 21:01:50.096760 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.096739 5855 scope.go:122] "RemoveContainer" containerID="b572ec05af989619c2dcce8b0e326b4383cd8f7ea35e356ad315cd61678e002e" Oct 30 21:01:50.107280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.107250 5855 scope.go:122] "RemoveContainer" containerID="5285df1a7b6da00aa8f0e1df3b26b9e59d58270f1eec782073dc4787f50db398" Oct 30 21:01:50.118086 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.118063 5855 scope.go:122] "RemoveContainer" containerID="24e0a733a6c43eb9b81650cf679452e34f113656a472eb1bdc6b01c64d46001f" Oct 30 21:01:50.129202 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:50.129179 5855 scope.go:122] "RemoveContainer" containerID="632edbd275b026c7240e4c28bc20acfb79081501a9303acf07c856624834cb38" Oct 30 21:01:51.070891 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:01:51.070855 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:02:00.922861 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:02:00.922811 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:02:01.023357 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:01.023322 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf"] Oct 30 21:02:01.027412 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:01.027383 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972fcca5-db8cf"] Oct 30 21:02:02.777964 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:02.777933 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15da1b9-a128-4942-a8c2-d33db7c92c1a" path="/var/lib/kubelet/pods/e15da1b9-a128-4942-a8c2-d33db7c92c1a/volumes" Oct 30 21:02:17.339149 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.339111 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv"] Oct 30 21:02:17.339641 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.339514 5855 cpu_manager.go:414] "containerMap: removing container" podUID="c63f9a62-dfb9-4a32-9d55-3fd851792cfe" containerName="revoke-keys" Oct 30 21:02:17.339641 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.339525 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63f9a62-dfb9-4a32-9d55-3fd851792cfe" containerName="revoke-keys" Oct 30 21:02:17.339641 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.339595 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63f9a62-dfb9-4a32-9d55-3fd851792cfe" containerName="revoke-keys" Oct 30 21:02:17.343167 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.343148 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.351574 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.351428 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:02:17.351574 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.351510 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:02:17.351574 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.351430 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:02:17.351574 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.351543 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:02:17.351964 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.351874 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv"] Oct 30 21:02:17.475417 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.475374 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66mn\" (UniqueName: \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.475605 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.475439 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.475605 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.475560 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.577154 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.577094 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66mn\" (UniqueName: \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.577370 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.577183 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.577370 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.577247 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.577975 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.577953 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.579803 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.579782 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.590050 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.589987 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66mn\" (UniqueName: \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\") pod \"maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.659140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.659104 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:17.940022 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:17.939991 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv"] Oct 30 21:02:17.941497 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:02:17.941471 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eaed47_f9e5_4f7c_935f_197f7f2ea5b9.slice/crio-dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf WatchSource:0}: Error finding container dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf: Status 404 returned error can't find the container with id dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf Oct 30 21:02:18.027340 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.027304 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n"] Oct 30 21:02:18.032385 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.032337 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc264cc3-dsj6n"] Oct 30 21:02:18.034632 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.034606 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk"] Oct 30 21:02:18.036391 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.036371 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b13184a-xtkgk"] Oct 30 21:02:18.778290 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.778257 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcdec09-632c-4c5f-9ba2-85de673420cc" path="/var/lib/kubelet/pods/5dcdec09-632c-4c5f-9ba2-85de673420cc/volumes" Oct 30 21:02:18.778685 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.778672 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafbb32f-9f7e-40ab-bd62-58b90f6c852f" path="/var/lib/kubelet/pods/bafbb32f-9f7e-40ab-bd62-58b90f6c852f/volumes" Oct 30 21:02:18.920311 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.920276 5855 generic.go:356] "Generic (PLEG): container finished" podID="f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" containerID="c930ac69260462a3d6cc548778abdfe8536f0f6fba0e8e93d48b0d0eb86f5a34" exitCode=0 Oct 30 21:02:18.920496 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.920360 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" event={"ID":"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9","Type":"ContainerDied","Data":"c930ac69260462a3d6cc548778abdfe8536f0f6fba0e8e93d48b0d0eb86f5a34"} Oct 30 21:02:18.920496 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:18.920408 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" event={"ID":"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9","Type":"ContainerStarted","Data":"dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf"} Oct 30 21:02:20.091208 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.091181 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:20.223931 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.223830 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\" (UniqueName: \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\") pod \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " Oct 30 21:02:20.224139 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.223987 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\") pod \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " Oct 30 21:02:20.224139 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.224096 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\") pod \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\" (UID: \"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9\") " Oct 30 21:02:20.224479 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.224454 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca" pod "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" (UID: "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:20.226428 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.226391 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca" pod "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" (UID: "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:20.226492 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.226396 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn" pod "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" (UID: "f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9"). InnerVolumeSpecName "kube-api-access-x66mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:20.325479 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.325433 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:20.325479 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.325469 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66mn\" (UniqueName: \"kubernetes.io/projected/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-kube-api-access-x66mn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:20.325479 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.325481 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:20.932702 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.932647 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" event={"ID":"f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9","Type":"ContainerDied","Data":"dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf"} Oct 30 21:02:20.932702 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.932700 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv" Oct 30 21:02:20.932917 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:20.932721 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dda5c4839631f84ce163798b10ce56a43a80a167d8de065a2369177c4db69bbf" Oct 30 21:02:22.157345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:22.157306 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 21:02:22.194964 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:22.194923 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 21:02:27.437993 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.437798 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 21:02:27.708739 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.704746 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk"] Oct 30 21:02:27.723425 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.709566 5855 cpu_manager.go:414] "containerMap: removing container" podUID="f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" containerName="revoke-keys" Oct 30 21:02:27.723425 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.721899 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" containerName="revoke-keys" Oct 30 21:02:27.723425 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.722155 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" containerName="revoke-keys" Oct 30 21:02:27.737959 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.737923 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk"] Oct 30 21:02:27.738166 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.738037 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:27.743442 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.743413 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:02:27.746465 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.743735 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:02:27.746681 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.743863 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:02:27.746824 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.743913 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:02:27.906633 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.906594 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:27.906887 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.906670 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlmzz\" (UniqueName: \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:27.906887 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:27.906829 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.007925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.007822 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlmzz\" (UniqueName: \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.007925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.007906 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.008162 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.007989 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.008601 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.008576 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.010593 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.010560 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.020486 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.020463 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlmzz\" (UniqueName: \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\") pod \"maas-api-revoke-sub-9972ce59-t8ztk\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.058478 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.058452 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:28.342743 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.342719 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk"] Oct 30 21:02:28.345480 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:02:28.345449 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65caec2e_b33f_476b_aa17_b43dca9a8e01.slice/crio-c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f WatchSource:0}: Error finding container c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f: Status 404 returned error can't find the container with id c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f Oct 30 21:02:28.982281 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.982191 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" event={"ID":"65caec2e-b33f-476b-aa17-b43dca9a8e01","Type":"ContainerStarted","Data":"ccf7a005163ddc1027f71e76626f284bbf3dd159432c8d182bdf4a60148f2a17"} Oct 30 21:02:28.982281 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:28.982247 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" event={"ID":"65caec2e-b33f-476b-aa17-b43dca9a8e01","Type":"ContainerStarted","Data":"c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f"} Oct 30 21:02:29.004269 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:29.004197 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" podStartSLOduration=2.004174584 podStartE2EDuration="2.004174584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:29.002019366 +0000 UTC m=+1362.360732123" watchObservedRunningTime="2026-10-30 21:02:29.004174584 +0000 UTC m=+1362.362887332" Oct 30 21:02:29.989524 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:29.989488 5855 generic.go:356] "Generic (PLEG): container finished" podID="65caec2e-b33f-476b-aa17-b43dca9a8e01" containerID="ccf7a005163ddc1027f71e76626f284bbf3dd159432c8d182bdf4a60148f2a17" exitCode=0 Oct 30 21:02:29.990116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:29.989580 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" event={"ID":"65caec2e-b33f-476b-aa17-b43dca9a8e01","Type":"ContainerDied","Data":"ccf7a005163ddc1027f71e76626f284bbf3dd159432c8d182bdf4a60148f2a17"} Oct 30 21:02:30.924330 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:02:30.924292 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:02:31.135754 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.135726 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:31.242404 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.242310 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\") pod \"65caec2e-b33f-476b-aa17-b43dca9a8e01\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " Oct 30 21:02:31.242699 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.242644 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\") pod \"65caec2e-b33f-476b-aa17-b43dca9a8e01\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " Oct 30 21:02:31.242810 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.242708 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca" pod "65caec2e-b33f-476b-aa17-b43dca9a8e01" (UID: "65caec2e-b33f-476b-aa17-b43dca9a8e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:31.242810 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.242760 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\" (UniqueName: \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\") pod \"65caec2e-b33f-476b-aa17-b43dca9a8e01\" (UID: \"65caec2e-b33f-476b-aa17-b43dca9a8e01\") " Oct 30 21:02:31.243298 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.243267 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:31.245850 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.245806 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca" pod "65caec2e-b33f-476b-aa17-b43dca9a8e01" (UID: "65caec2e-b33f-476b-aa17-b43dca9a8e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:31.245955 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.245874 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz" pod "65caec2e-b33f-476b-aa17-b43dca9a8e01" (UID: "65caec2e-b33f-476b-aa17-b43dca9a8e01"). InnerVolumeSpecName "kube-api-access-jlmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:31.344776 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.344740 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65caec2e-b33f-476b-aa17-b43dca9a8e01-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:31.344776 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:31.344768 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlmzz\" (UniqueName: \"kubernetes.io/projected/65caec2e-b33f-476b-aa17-b43dca9a8e01-kube-api-access-jlmzz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:32.012462 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:32.012424 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" event={"ID":"65caec2e-b33f-476b-aa17-b43dca9a8e01","Type":"ContainerDied","Data":"c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f"} Oct 30 21:02:32.012462 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:32.012452 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk" Oct 30 21:02:32.012707 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:32.012513 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2665bf1cf3cbec8ddc3bedbda20cc8e45d3e650971f5511a07dfda78210b95f" Oct 30 21:02:32.439389 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:32.439291 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" containerID="cri-o://0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082" gracePeriod=25 Oct 30 21:02:36.547087 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:36.547051 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:02:37.178480 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.178439 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" containerID="cri-o://0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a" gracePeriod=45 Oct 30 21:02:37.216596 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.216554 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" containerID="cri-o://f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2" gracePeriod=45 Oct 30 21:02:37.475935 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.475900 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 21:02:37.508585 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508549 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.508779 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508689 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\" (UniqueName: \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.508779 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508760 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.508908 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508817 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.508908 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508863 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.509010 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.508918 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\") pod \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\" (UID: \"904d6f64-339e-4a4c-b265-9bb2ac9a63a1\") " Oct 30 21:02:37.509461 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.509434 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.509620 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.509602 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.512898 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.512863 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "kube-api-access-5gpqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:37.513115 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.513092 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.517140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.517107 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.517262 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.517163 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs" pod "904d6f64-339e-4a4c-b265-9bb2ac9a63a1" (UID: "904d6f64-339e-4a4c-b265-9bb2ac9a63a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:37.522206 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.522181 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 21:02:37.609453 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609417 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609506 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609569 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609603 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\" (UniqueName: \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609648 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609708 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\") pod \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\" (UID: \"916e593c-f5ee-47a9-bcc7-7167382c4d4b\") " Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609643 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.609948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.609824 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610017 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610040 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610063 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610079 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpqd\" (UniqueName: \"kubernetes.io/projected/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-kube-api-access-5gpqd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610093 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610106 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610118 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610130 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.610345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.610143 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/904d6f64-339e-4a4c-b265-9bb2ac9a63a1-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.611796 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.611773 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.611957 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.611935 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "kube-api-access-b5sdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:37.612036 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.612017 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs" pod "916e593c-f5ee-47a9-bcc7-7167382c4d4b" (UID: "916e593c-f5ee-47a9-bcc7-7167382c4d4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:37.711470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.711389 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.711470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.711439 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5sdk\" (UniqueName: \"kubernetes.io/projected/916e593c-f5ee-47a9-bcc7-7167382c4d4b-kube-api-access-b5sdk\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.711470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.711450 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/916e593c-f5ee-47a9-bcc7-7167382c4d4b-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.711470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:37.711459 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/916e593c-f5ee-47a9-bcc7-7167382c4d4b-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:38.049048 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.049013 5855 generic.go:356] "Generic (PLEG): container finished" podID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerID="f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2" exitCode=0 Oct 30 21:02:38.049227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.049073 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" Oct 30 21:02:38.049227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.049096 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" event={"ID":"916e593c-f5ee-47a9-bcc7-7167382c4d4b","Type":"ContainerDied","Data":"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2"} Oct 30 21:02:38.049227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.049137 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb" event={"ID":"916e593c-f5ee-47a9-bcc7-7167382c4d4b","Type":"ContainerDied","Data":"68386f544bf815e1772cf5be236a2738bfdaa5ccca4198895c33af44a311f889"} Oct 30 21:02:38.049227 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.049155 5855 scope.go:122] "RemoveContainer" containerID="f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2" Oct 30 21:02:38.050710 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.050678 5855 generic.go:356] "Generic (PLEG): container finished" podID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerID="0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a" exitCode=0 Oct 30 21:02:38.050845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.050745 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" Oct 30 21:02:38.050845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.050777 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" event={"ID":"904d6f64-339e-4a4c-b265-9bb2ac9a63a1","Type":"ContainerDied","Data":"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a"} Oct 30 21:02:38.050845 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.050816 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g" event={"ID":"904d6f64-339e-4a4c-b265-9bb2ac9a63a1","Type":"ContainerDied","Data":"eb9cd6f66d414cd09e5615ec29c1627a613e19aaf51fe5efe8f627d1a9d53ca9"} Oct 30 21:02:38.061436 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.061415 5855 scope.go:122] "RemoveContainer" containerID="f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2" Oct 30 21:02:38.062575 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:02:38.061781 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2\": container with ID starting with f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2 not found: ID does not exist" containerID="f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2" Oct 30 21:02:38.062575 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.061821 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2"} err="failed to get container status \"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2\": rpc error: code = NotFound desc = could not find container \"f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2\": container with ID starting with f062f78e034191a3d26f511ea64e2faaf847904bc4044a218cf4c7cef341a7b2 not found: ID does not exist" Oct 30 21:02:38.062575 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.061847 5855 scope.go:122] "RemoveContainer" containerID="0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a" Oct 30 21:02:38.073060 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.073036 5855 scope.go:122] "RemoveContainer" containerID="0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a" Oct 30 21:02:38.073413 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:02:38.073386 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a\": container with ID starting with 0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a not found: ID does not exist" containerID="0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a" Oct 30 21:02:38.073515 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.073421 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a"} err="failed to get container status \"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a\": rpc error: code = NotFound desc = could not find container \"0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a\": container with ID starting with 0a3971a4795abfe4602f356e13dc0e7677409e97f2c25fe6f0c3a1b25a0bc43a not found: ID does not exist" Oct 30 21:02:38.074147 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.074125 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 21:02:38.082462 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.082439 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/simulator-w4-f2ed1e-kserve-77859ccd9d-nsrsb"] Oct 30 21:02:38.091244 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.091204 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 21:02:38.096198 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.096168 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-2-w4-f2ed1e-kserve-65c9894669-mgx9g"] Oct 30 21:02:38.779037 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.779003 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" path="/var/lib/kubelet/pods/904d6f64-339e-4a4c-b265-9bb2ac9a63a1/volumes" Oct 30 21:02:38.779455 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:38.779439 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" path="/var/lib/kubelet/pods/916e593c-f5ee-47a9-bcc7-7167382c4d4b/volumes" Oct 30 21:02:39.945397 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:39.945363 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:02:42.917046 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:42.917004 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv"] Oct 30 21:02:42.921245 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:42.921217 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f2ed1e-tl9pv"] Oct 30 21:02:44.778228 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:44.778195 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9" path="/var/lib/kubelet/pods/f8eaed47-f9e5-4f7c-935f-197f7f2ea5b9/volumes" Oct 30 21:02:45.983670 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:45.983604 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:02:46.547039 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:46.547004 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:02:49.988992 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:49.988951 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 21:02:49.989979 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:49.989923 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" containerID="cri-o://c854f84492559e56279dd27563324f24bba3a5b51de30b84c7d70d9a7da85d32" gracePeriod=30 Oct 30 21:02:50.232469 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:50.232421 5855 scope.go:122] "RemoveContainer" containerID="ff5795d27b99f0b53fe3e7698ba78d1cde33e4251b84566def5f415b8f13872a" Oct 30 21:02:50.241907 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:50.241880 5855 scope.go:122] "RemoveContainer" containerID="5bccff62b1b943f526028f19eeb025f027e9d4b245e6d44681e6093c8eedef0d" Oct 30 21:02:50.252985 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:50.252964 5855 scope.go:122] "RemoveContainer" containerID="f495fe9df4312f8454c9a0a217a399c9ff9a78b2c130ac15a200e437cc81c288" Oct 30 21:02:50.263306 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:50.263281 5855 scope.go:122] "RemoveContainer" containerID="aef99de9bb73142ddde3ee24741217c2552bf55bf433785ebbd55c7e96d723dc" Oct 30 21:02:50.286555 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:50.286527 5855 scope.go:122] "RemoveContainer" containerID="9e71d773e8e93db4521eedb6b73bd7ce3e7786f44dadf5c32a124a4949eeed0d" Oct 30 21:02:52.687815 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.687783 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 21:02:52.770987 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.770901 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\") pod \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " Oct 30 21:02:52.770987 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.770972 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\" (UniqueName: \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\") pod \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\" (UID: \"d0ff0d13-613d-4e61-9309-717c5dd63fdc\") " Oct 30 21:02:52.771284 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.771257 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume" pod "d0ff0d13-613d-4e61-9309-717c5dd63fdc" (UID: "d0ff0d13-613d-4e61-9309-717c5dd63fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:52.773223 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.773202 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm" pod "d0ff0d13-613d-4e61-9309-717c5dd63fdc" (UID: "d0ff0d13-613d-4e61-9309-717c5dd63fdc"). InnerVolumeSpecName "kube-api-access-9pbjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:52.871850 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.871807 5855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff0d13-613d-4e61-9309-717c5dd63fdc-plugins-config-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:52.871850 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:52.871843 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbjm\" (UniqueName: \"kubernetes.io/projected/d0ff0d13-613d-4e61-9309-717c5dd63fdc-kube-api-access-9pbjm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:53.158833 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.158796 5855 generic.go:356] "Generic (PLEG): container finished" podID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerID="0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082" exitCode=0 Oct 30 21:02:53.159021 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.158848 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" Oct 30 21:02:53.159021 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.158878 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" event={"ID":"d0ff0d13-613d-4e61-9309-717c5dd63fdc","Type":"ContainerDied","Data":"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082"} Oct 30 21:02:53.159021 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.158939 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw" event={"ID":"d0ff0d13-613d-4e61-9309-717c5dd63fdc","Type":"ContainerDied","Data":"2acad65ac71afa8860cf19d8e513236358ecb02240510c1c5eef2ebab4c0f957"} Oct 30 21:02:53.159021 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.158963 5855 scope.go:122] "RemoveContainer" containerID="0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082" Oct 30 21:02:53.168460 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.168443 5855 scope.go:122] "RemoveContainer" containerID="0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082" Oct 30 21:02:53.168814 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:02:53.168793 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082\": container with ID starting with 0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082 not found: ID does not exist" containerID="0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082" Oct 30 21:02:53.168871 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.168824 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082"} err="failed to get container status \"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082\": rpc error: code = NotFound desc = could not find container \"0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082\": container with ID starting with 0084b3c01ee113d31d909e99134836a764642f4fc9985014c97795522f6da082 not found: ID does not exist" Oct 30 21:02:53.177542 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.177504 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 21:02:53.183503 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:53.183469 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f2ed1e-784697bfcb-592bw"] Oct 30 21:02:54.780267 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:54.780216 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" path="/var/lib/kubelet/pods/d0ff0d13-613d-4e61-9309-717c5dd63fdc/volumes" Oct 30 21:02:54.971298 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:54.971260 5855 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:02:54.971462 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:54.971318 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:02:55.177774 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.177739 5855 generic.go:356] "Generic (PLEG): container finished" podID="af9eae5e-4062-4950-af86-4cf931762420" containerID="c854f84492559e56279dd27563324f24bba3a5b51de30b84c7d70d9a7da85d32" exitCode=0 Oct 30 21:02:55.177948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.177808 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" event={"ID":"af9eae5e-4062-4950-af86-4cf931762420","Type":"ContainerDied","Data":"c854f84492559e56279dd27563324f24bba3a5b51de30b84c7d70d9a7da85d32"} Oct 30 21:02:55.249027 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.248996 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 21:02:55.395278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395188 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395231 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395264 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395539 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395322 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395539 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395354 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395539 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395411 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395539 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395483 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.395921 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395539 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.396002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395964 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\") pod \"af9eae5e-4062-4950-af86-4cf931762420\" (UID: \"af9eae5e-4062-4950-af86-4cf931762420\") " Oct 30 21:02:55.396002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395824 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.396111 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395824 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.396111 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.395842 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.396387 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396121 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:55.396387 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396197 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.396387 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396366 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.396387 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396381 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.396600 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396390 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.396600 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396399 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9eae5e-4062-4950-af86-4cf931762420-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.396600 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.396407 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.397983 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.397951 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "kube-api-access-ncx2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.398275 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.398251 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:55.398275 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.398264 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.398465 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.398322 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token" pod "af9eae5e-4062-4950-af86-4cf931762420" (UID: "af9eae5e-4062-4950-af86-4cf931762420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.497002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.496948 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9eae5e-4062-4950-af86-4cf931762420-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.497002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.496994 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9eae5e-4062-4950-af86-4cf931762420-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.497002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.497004 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.497002 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:55.497012 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncx2m\" (UniqueName: \"kubernetes.io/projected/af9eae5e-4062-4950-af86-4cf931762420-kube-api-access-ncx2m\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:56.184405 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.184377 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" Oct 30 21:02:56.184405 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.184389 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt" event={"ID":"af9eae5e-4062-4950-af86-4cf931762420","Type":"ContainerDied","Data":"63799ba838839a4cc4171034fe781a38adcf2ce514b37c77468dd54318a2f93d"} Oct 30 21:02:56.184937 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.184449 5855 scope.go:122] "RemoveContainer" containerID="c854f84492559e56279dd27563324f24bba3a5b51de30b84c7d70d9a7da85d32" Oct 30 21:02:56.208025 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.207995 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 21:02:56.211873 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.211849 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-ph6rt"] Oct 30 21:02:56.777504 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:56.777468 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9eae5e-4062-4950-af86-4cf931762420" path="/var/lib/kubelet/pods/af9eae5e-4062-4950-af86-4cf931762420/volumes" Oct 30 21:02:59.033159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:59.033103 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh"] Oct 30 21:02:59.035161 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:59.035124 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg"] Oct 30 21:02:59.041777 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:59.041746 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121ae6d-dpkkh"] Oct 30 21:02:59.043162 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:02:59.043137 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bd1c62c-h7kdg"] Oct 30 21:03:00.643893 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:00.643858 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 21:03:00.644427 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:00.644118 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerName="discovery" containerID="cri-o://5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3" gracePeriod=30 Oct 30 21:03:00.779595 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:00.779553 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543c1b58-56e7-4c66-a866-4e132fc24c08" path="/var/lib/kubelet/pods/543c1b58-56e7-4c66-a866-4e132fc24c08/volumes" Oct 30 21:03:00.780077 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:00.780062 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80ded81-5681-4a8c-a20a-d45a383d423b" path="/var/lib/kubelet/pods/d80ded81-5681-4a8c-a20a-d45a383d423b/volumes" Oct 30 21:03:00.915347 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:00.915311 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 21:03:00.932713 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:03:00.932641 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:03:01.022366 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.022330 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl"] Oct 30 21:03:01.025290 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.025264 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe5ffbca-wsxcl"] Oct 30 21:03:01.058000 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.057968 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058214 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058036 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058214 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058069 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058214 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058155 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058214 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058212 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058428 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058270 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.058428 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058305 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\") pod \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\" (UID: \"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6\") " Oct 30 21:03:01.059008 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.058776 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:01.061925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.061744 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:01.062092 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.062065 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:01.062092 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.062071 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:01.062233 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.062153 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:01.062233 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.062200 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:01.062334 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.062277 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb" pod "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" (UID: "d384c4f9-5e62-4ad7-8b55-5d2eac4981b6"). InnerVolumeSpecName "kube-api-access-r4hcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160044 5855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-dns-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160075 5855 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-kubeconfig\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160086 5855 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-csr-ca-configmap\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160098 5855 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-cacerts\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160107 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4hcb\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-kube-api-access-r4hcb\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160116 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.160124 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.160125 5855 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6-local-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.212895 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.212860 5855 generic.go:356] "Generic (PLEG): container finished" podID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerID="5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3" exitCode=0 Oct 30 21:03:01.213089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.212917 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" Oct 30 21:03:01.213089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.212940 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" event={"ID":"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6","Type":"ContainerDied","Data":"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3"} Oct 30 21:03:01.213089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.212992 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv" event={"ID":"d384c4f9-5e62-4ad7-8b55-5d2eac4981b6","Type":"ContainerDied","Data":"5b6fa019d79749a46e544337726499934085b757266350ae8451e8d7022796ba"} Oct 30 21:03:01.213089 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.213012 5855 scope.go:122] "RemoveContainer" containerID="5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3" Oct 30 21:03:01.222965 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.222941 5855 scope.go:122] "RemoveContainer" containerID="5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3" Oct 30 21:03:01.223247 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:03:01.223227 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3\": container with ID starting with 5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3 not found: ID does not exist" containerID="5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3" Oct 30 21:03:01.223299 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.223257 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3"} err="failed to get container status \"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3\": rpc error: code = NotFound desc = could not find container \"5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3\": container with ID starting with 5e8517bcb7f692272a09a378bc27aa80f005f7591e360c6f245cfecb15001ce3 not found: ID does not exist" Oct 30 21:03:01.237281 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.237249 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 21:03:01.241643 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:01.241620 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b6fwv"] Oct 30 21:03:02.778548 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:02.778508 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" path="/var/lib/kubelet/pods/d384c4f9-5e62-4ad7-8b55-5d2eac4981b6/volumes" Oct 30 21:03:02.778952 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:02.778892 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe019915-e577-45b7-96ae-ece4e2c1718b" path="/var/lib/kubelet/pods/fe019915-e577-45b7-96ae-ece4e2c1718b/volumes" Oct 30 21:03:10.023607 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:10.023573 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8"] Oct 30 21:03:10.026752 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:10.026725 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace53c0-qnfm8"] Oct 30 21:03:10.777481 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:10.777446 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525da2a4-998a-4f8b-a5d7-83d51dde8c31" path="/var/lib/kubelet/pods/525da2a4-998a-4f8b-a5d7-83d51dde8c31/volumes" Oct 30 21:03:30.925327 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:03:30.925286 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:03:49.036731 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:49.036693 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d"] Oct 30 21:03:49.039682 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:49.039626 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e17370-9gb2d"] Oct 30 21:03:50.365447 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.365403 5855 scope.go:122] "RemoveContainer" containerID="ff55308c78fd244f3e14ed628413dad9d10c8f95853e9e7745f22b9a799e7f87" Oct 30 21:03:50.374627 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.374602 5855 scope.go:122] "RemoveContainer" containerID="43d6e47a3d9554c0a4f343a99d5ef6246d2142794049298b24cf939d6c909aa4" Oct 30 21:03:50.382827 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.382807 5855 scope.go:122] "RemoveContainer" containerID="a66e570ff4ad6d2cb831facd9325aec1c10774d89a7404b46b742e29d845e22e" Oct 30 21:03:50.391743 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.391717 5855 scope.go:122] "RemoveContainer" containerID="2ca81756361fde404eaa27edcd21127711bfb03c0d03518cf36af6824222b5bf" Oct 30 21:03:50.400451 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.400431 5855 scope.go:122] "RemoveContainer" containerID="e0dd2760d544ca918d286020b1f950e185d83bad0df5ed30d854753af969519d" Oct 30 21:03:50.408600 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.408575 5855 scope.go:122] "RemoveContainer" containerID="527f7d419d4348e1328c49ee6780c3b61857931720ac31bcdb222081d411de99" Oct 30 21:03:50.417809 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.417783 5855 scope.go:122] "RemoveContainer" containerID="7458b56b615022431eeccd814a19de2a9dd758773c4b9d263fb5e86ced383762" Oct 30 21:03:50.426566 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.426537 5855 scope.go:122] "RemoveContainer" containerID="1dc2b173d0743deacc96a05a0822a4f7faf0cd441baf18f79312f5caf49b2752" Oct 30 21:03:50.777753 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:03:50.777674 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201f8433-1c4b-40db-b1ad-4e730947f2b5" path="/var/lib/kubelet/pods/201f8433-1c4b-40db-b1ad-4e730947f2b5/volumes" Oct 30 21:04:00.923944 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:04:00.923858 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:04:16.764781 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:16.764749 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:04:30.925348 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:04:30.925310 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:04:35.137514 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:35.137474 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:04:39.331824 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:39.331775 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:04:48.027497 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:48.027456 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g"] Oct 30 21:04:48.031197 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:48.031170 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60515392-rz69g"] Oct 30 21:04:48.777347 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:48.777297 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ea5d42-f76c-48c4-b812-57096189e0f3" path="/var/lib/kubelet/pods/46ea5d42-f76c-48c4-b812-57096189e0f3/volumes" Oct 30 21:04:50.489140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.489104 5855 scope.go:122] "RemoveContainer" containerID="34b8ad6c1ad981c71a38b162bb44e5067f7ab69077620a6a75e9fcb70d869803" Oct 30 21:04:50.497972 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.497949 5855 scope.go:122] "RemoveContainer" containerID="0ed28654bb19ef4cf62b228a980544ff0cedd96c15304c0adb9cf1da70aa8c12" Oct 30 21:04:50.506957 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.506934 5855 scope.go:122] "RemoveContainer" containerID="5fad1dfd2189542806e4042504ee1037bf888193bc76287248d182a304e95919" Oct 30 21:04:50.516030 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.516006 5855 scope.go:122] "RemoveContainer" containerID="1bb82061c900d652e937a971b23ce50b75aa16b961c8d4a864128a7c9cf65962" Oct 30 21:04:50.524400 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.524372 5855 scope.go:122] "RemoveContainer" containerID="3bd89aaa889932b7996ec2c7fb73ea60c9948fc7e56702bbd0f613a4abd96ec5" Oct 30 21:04:50.535025 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:50.535002 5855 scope.go:122] "RemoveContainer" containerID="1c1a910469b562ecc37a80c3a53869319893097b79c8449aa3ee26e09b1a43cb" Oct 30 21:04:53.769159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:04:53.769124 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:05:00.923456 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:05:00.923412 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:05:06.148906 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:06.148862 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:05:18.992686 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:18.986806 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:05:18.992686 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:18.986924 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:05:30.924851 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:05:30.924810 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:05:38.026563 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:38.026527 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z"] Oct 30 21:05:38.034167 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:38.034055 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7adbb7-6ng2z"] Oct 30 21:05:38.451465 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:38.451432 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:05:38.782912 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:38.782801 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fceb8e-1b0e-4aa3-81d7-236f42183ee2" path="/var/lib/kubelet/pods/52fceb8e-1b0e-4aa3-81d7-236f42183ee2/volumes" Oct 30 21:05:48.532167 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:48.532123 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:05:50.572973 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:50.572942 5855 scope.go:122] "RemoveContainer" containerID="60ce51896d3becf10b4ae5ed9b548ebcf251efbb4e2363c6341791d9e937b992" Oct 30 21:05:50.582977 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:50.582915 5855 scope.go:122] "RemoveContainer" containerID="72b4c736005dffb1fec62c8598bfd18992828712217d5ba2f63dd31671c36eda" Oct 30 21:05:53.469151 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:05:53.469086 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:00.924448 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:06:00.924407 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:06:09.460961 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:09.460920 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:13.302675 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:13.302567 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:27.738059 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:27.738024 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:30.933354 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:06:30.933299 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:06:33.255443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:33.255406 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:46.477348 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:46.477309 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:50.616371 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:50.616325 5855 scope.go:122] "RemoveContainer" containerID="8ef6f6150f7861bb116e9f939081cf86c4040cc9e5a8d4879af03128127c1b78" Oct 30 21:06:50.664009 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:50.663869 5855 scope.go:122] "RemoveContainer" containerID="2a7e776be0a8a372ab605ee18ebf31205ac1629351194780e1777d164f4b7495" Oct 30 21:06:50.730558 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:50.729806 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:06:50.809850 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:50.809805 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-ph82z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:06:50.810230 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:06:50.810193 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:07:00.924176 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:07:00.924087 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:07:14.355613 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:14.355578 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:07:24.981024 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:24.980977 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:07:29.939530 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:29.939487 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:07:30.148748 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:30.148708 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:07:30.149078 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:30.149046 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:07:30.924169 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:07:30.924132 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:07:32.023435 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:32.023401 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk"] Oct 30 21:07:32.025877 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:32.025849 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9972ce59-t8ztk"] Oct 30 21:07:32.776795 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:32.776763 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65caec2e-b33f-476b-aa17-b43dca9a8e01" path="/var/lib/kubelet/pods/65caec2e-b33f-476b-aa17-b43dca9a8e01/volumes" Oct 30 21:07:46.971764 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:46.971723 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:07:50.757578 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:50.757543 5855 scope.go:122] "RemoveContainer" containerID="5b162f37409fb8765663b6bef097167afb0465333392e926808f9b707a7e2de9" Oct 30 21:07:50.950525 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:07:50.950480 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:08:00.924832 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:08:00.924790 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:08:06.944804 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:08:06.944763 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:08:25.455443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:08:25.455357 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:08:30.923212 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:08:30.923174 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:08:35.577628 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:08:35.577483 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:08:50.777121 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:08:50.777083 5855 scope.go:122] "RemoveContainer" containerID="c930ac69260462a3d6cc548778abdfe8536f0f6fba0e8e93d48b0d0eb86f5a34" Oct 30 21:08:50.785746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:08:50.785716 5855 scope.go:122] "RemoveContainer" containerID="ccf7a005163ddc1027f71e76626f284bbf3dd159432c8d182bdf4a60148f2a17" Oct 30 21:09:00.926481 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:09:00.926439 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:09:02.163171 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:02.163134 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:09:03.964882 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:03.964830 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:09:03.965669 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:03.964908 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:06.786408 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:06.786368 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:09:19.570170 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:19.570130 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:09:29.776006 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:29.775966 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:09:30.928710 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:09:30.928627 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:09:46.107707 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:09:46.107643 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:10:00.925847 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:10:00.925764 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:10:10.358008 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:10:10.357963 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:10:21.747879 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:10:21.747841 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:10:30.922915 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:10:30.922877 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:10:40.043901 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:10:40.043862 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:10:44.841997 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:10:44.841954 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:11:00.752219 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:00.752183 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 21:11:00.752724 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:00.752392 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" podUID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" containerName="manager" containerID="cri-o://f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535" gracePeriod=10 Oct 30 21:11:00.927585 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:11:00.927545 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:11:01.007484 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.007411 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 21:11:01.098243 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.098205 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\") pod \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " Oct 30 21:11:01.098468 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.098288 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\" (UniqueName: \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\") pod \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\" (UID: \"0c3aee86-ff23-4995-b442-2302bb1ee6ba\") " Oct 30 21:11:01.098719 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.098689 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume" pod "0c3aee86-ff23-4995-b442-2302bb1ee6ba" (UID: "0c3aee86-ff23-4995-b442-2302bb1ee6ba"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:11:01.100832 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.100803 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn" pod "0c3aee86-ff23-4995-b442-2302bb1ee6ba" (UID: "0c3aee86-ff23-4995-b442-2302bb1ee6ba"). InnerVolumeSpecName "kube-api-access-l9jtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:11:01.199250 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.199209 5855 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c3aee86-ff23-4995-b442-2302bb1ee6ba-extensions-socket-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:11:01.199250 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.199245 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jtn\" (UniqueName: \"kubernetes.io/projected/0c3aee86-ff23-4995-b442-2302bb1ee6ba-kube-api-access-l9jtn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:11:01.836017 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.835975 5855 generic.go:356] "Generic (PLEG): container finished" podID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" containerID="f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535" exitCode=0 Oct 30 21:11:01.836457 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.836046 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" Oct 30 21:11:01.836457 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.836057 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" event={"ID":"0c3aee86-ff23-4995-b442-2302bb1ee6ba","Type":"ContainerDied","Data":"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535"} Oct 30 21:11:01.836457 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.836114 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r" event={"ID":"0c3aee86-ff23-4995-b442-2302bb1ee6ba","Type":"ContainerDied","Data":"df37d2662e2a0e07ba8952d300b6a27f06b5999c8178a14fd83ffd465405f420"} Oct 30 21:11:01.836457 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.836135 5855 scope.go:122] "RemoveContainer" containerID="f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535" Oct 30 21:11:01.847397 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.847379 5855 scope.go:122] "RemoveContainer" containerID="f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535" Oct 30 21:11:01.847683 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:11:01.847646 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535\": container with ID starting with f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535 not found: ID does not exist" containerID="f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535" Oct 30 21:11:01.847748 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.847693 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535"} err="failed to get container status \"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535\": rpc error: code = NotFound desc = could not find container \"f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535\": container with ID starting with f381e1746684239f814476ded1b837e53664df152f34308bf8d052fa569bd535 not found: ID does not exist" Oct 30 21:11:01.859948 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.859927 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 21:11:01.866852 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:01.866831 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-dnn7r"] Oct 30 21:11:02.777037 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:11:02.777005 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" path="/var/lib/kubelet/pods/0c3aee86-ff23-4995-b442-2302bb1ee6ba/volumes" Oct 30 21:11:30.925264 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:11:30.925166 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:12:00.924442 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:12:00.924400 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:12:20.318852 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:12:20.318812 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:12:30.923970 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:12:30.923924 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:12:34.505896 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:12:34.505859 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:12:45.791157 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:12:45.791122 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:13:00.539936 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:00.539851 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:13:00.926063 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:13:00.926016 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:13:29.402063 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:29.402029 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:13:30.924370 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:13:30.924330 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:13:42.419914 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.419871 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:13:42.420448 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420427 5855 cpu_manager.go:414] "containerMap: removing container" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" Oct 30 21:13:42.420512 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420452 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" Oct 30 21:13:42.420512 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420487 5855 cpu_manager.go:414] "containerMap: removing container" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" Oct 30 21:13:42.420512 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420496 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420516 5855 cpu_manager.go:414] "containerMap: removing container" podUID="65caec2e-b33f-476b-aa17-b43dca9a8e01" containerName="revoke-keys" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420525 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65caec2e-b33f-476b-aa17-b43dca9a8e01" containerName="revoke-keys" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420544 5855 cpu_manager.go:414] "containerMap: removing container" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420552 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420563 5855 cpu_manager.go:414] "containerMap: removing container" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420570 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420581 5855 cpu_manager.go:414] "containerMap: removing container" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerName="discovery" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420588 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerName="discovery" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420599 5855 cpu_manager.go:414] "containerMap: removing container" podUID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" containerName="manager" Oct 30 21:13:42.420684 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420605 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" containerName="manager" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420728 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="65caec2e-b33f-476b-aa17-b43dca9a8e01" containerName="revoke-keys" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420742 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9eae5e-4062-4950-af86-4cf931762420" containerName="istio-proxy" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420747 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="904d6f64-339e-4a4c-b265-9bb2ac9a63a1" containerName="main" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420753 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d384c4f9-5e62-4ad7-8b55-5d2eac4981b6" containerName="discovery" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420758 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c3aee86-ff23-4995-b442-2302bb1ee6ba" containerName="manager" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420765 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="916e593c-f5ee-47a9-bcc7-7167382c4d4b" containerName="main" Oct 30 21:13:42.421140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.420773 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ff0d13-613d-4e61-9309-717c5dd63fdc" containerName="payload-processing" Oct 30 21:13:42.423962 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.423942 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.436306 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.436278 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-30d5ac\"/\"default-dockercfg-wbkqk\"" Oct 30 21:13:42.436502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.436381 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-30d5ac\"/\"openshift-service-ca.crt\"" Oct 30 21:13:42.436502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.436475 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-30d5ac\"/\"kube-root-ca.crt\"" Oct 30 21:13:42.436680 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.436626 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-30d5ac\"/\"simulator-single-main-30d5ac-kserve-self-signed-certs\"" Oct 30 21:13:42.437107 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.437086 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:13:42.536274 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536230 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.536494 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536288 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.536494 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536370 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.536494 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536460 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn9pk\" (UniqueName: \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.536648 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536526 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.536648 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.536581 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637289 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637247 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637486 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637304 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637486 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637430 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637486 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637475 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn9pk\" (UniqueName: \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637620 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637515 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637620 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637558 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637776 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637751 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637838 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637819 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.637895 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.637855 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.640179 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.640155 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.640354 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.640337 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.656150 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.656115 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn9pk\" (UniqueName: \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\") pod \"simulator-single-main-30d5ac-kserve-76774599fd-9cfds\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:42.743818 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:42.743721 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:43.038730 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:43.034864 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:13:43.586146 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:43.586112 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" event={"ID":"b0713a19-aeda-4e77-953a-398636610739","Type":"ContainerStarted","Data":"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6"} Oct 30 21:13:43.586557 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:43.586159 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" event={"ID":"b0713a19-aeda-4e77-953a-398636610739","Type":"ContainerStarted","Data":"1beb943c876628e32a18de2e8345fa55c6dbb9f845ce94b346756b4102061c07"} Oct 30 21:13:43.609533 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:43.609477 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" podStartSLOduration=1.609463888 podStartE2EDuration="1.609463888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:43.608576233 +0000 UTC m=+2036.967289002" watchObservedRunningTime="2026-10-30 21:13:43.609463888 +0000 UTC m=+2036.968176625" Oct 30 21:13:43.744481 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:43.744445 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:44.158220 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:44.158180 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:13:52.744846 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:52.744806 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:52.757667 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:52.757636 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:13:53.651339 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:13:53.651307 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:14:00.924506 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:14:00.924467 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:14:10.525715 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:10.525669 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:14:22.329152 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.329108 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6"] Oct 30 21:14:22.333075 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.333049 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.341339 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.341305 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:14:22.343890 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.343860 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:14:22.343890 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.343860 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:14:22.345419 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.345393 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6"] Oct 30 21:14:22.345555 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.345396 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:14:22.434305 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.434270 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.434509 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.434347 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.434509 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.434473 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkffm\" (UniqueName: \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.535191 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.535159 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.535411 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.535240 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.535411 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.535273 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkffm\" (UniqueName: \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.535912 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.535888 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.537858 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.537837 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.550846 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.550817 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkffm\" (UniqueName: \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\") pod \"maas-api-revoke-sub-4ad14ae1-686x6\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.647207 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.647114 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:22.924708 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:22.924681 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6"] Oct 30 21:14:22.927437 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:14:22.927406 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545a95c9_c8f1_4b6d_a42f_e8863058b58e.slice/crio-e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e WatchSource:0}: Error finding container e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e: Status 404 returned error can't find the container with id e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e Oct 30 21:14:23.780595 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:23.780562 5855 generic.go:356] "Generic (PLEG): container finished" podID="545a95c9-c8f1-4b6d-a42f-e8863058b58e" containerID="000a7ab0a553a11ff483419c7be11358dc5bff3f7ef5455a439e26e588e17e34" exitCode=0 Oct 30 21:14:23.781113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:23.780612 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" event={"ID":"545a95c9-c8f1-4b6d-a42f-e8863058b58e","Type":"ContainerDied","Data":"000a7ab0a553a11ff483419c7be11358dc5bff3f7ef5455a439e26e588e17e34"} Oct 30 21:14:23.781113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:23.780679 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" event={"ID":"545a95c9-c8f1-4b6d-a42f-e8863058b58e","Type":"ContainerStarted","Data":"e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e"} Oct 30 21:14:24.917152 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.917112 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:24.958127 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.958092 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\" (UniqueName: \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\") pod \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " Oct 30 21:14:24.958300 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.958150 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\") pod \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " Oct 30 21:14:24.958300 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.958266 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\") pod \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\" (UID: \"545a95c9-c8f1-4b6d-a42f-e8863058b58e\") " Oct 30 21:14:24.958505 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.958483 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca" pod "545a95c9-c8f1-4b6d-a42f-e8863058b58e" (UID: "545a95c9-c8f1-4b6d-a42f-e8863058b58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:24.958616 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.958600 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:24.960426 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.960402 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca" pod "545a95c9-c8f1-4b6d-a42f-e8863058b58e" (UID: "545a95c9-c8f1-4b6d-a42f-e8863058b58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:24.960500 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:24.960488 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm" pod "545a95c9-c8f1-4b6d-a42f-e8863058b58e" (UID: "545a95c9-c8f1-4b6d-a42f-e8863058b58e"). InnerVolumeSpecName "kube-api-access-fkffm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:25.059110 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:25.059068 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkffm\" (UniqueName: \"kubernetes.io/projected/545a95c9-c8f1-4b6d-a42f-e8863058b58e-kube-api-access-fkffm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:25.059110 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:25.059110 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/545a95c9-c8f1-4b6d-a42f-e8863058b58e-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:25.789995 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:25.789964 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" Oct 30 21:14:25.790202 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:25.789969 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6" event={"ID":"545a95c9-c8f1-4b6d-a42f-e8863058b58e","Type":"ContainerDied","Data":"e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e"} Oct 30 21:14:25.790202 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:25.790090 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e011336a8bc3e12432a53b88f9a9dc97cfa3ee3c2285dd6dcd5456151b92553e" Oct 30 21:14:30.925345 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:14:30.925303 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:14:33.402815 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:33.402778 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:14:38.130705 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:38.130649 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:14:43.245513 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.245469 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85"] Oct 30 21:14:43.246043 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.246013 5855 cpu_manager.go:414] "containerMap: removing container" podUID="545a95c9-c8f1-4b6d-a42f-e8863058b58e" containerName="revoke-keys" Oct 30 21:14:43.246043 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.246032 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545a95c9-c8f1-4b6d-a42f-e8863058b58e" containerName="revoke-keys" Oct 30 21:14:43.246159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.246125 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="545a95c9-c8f1-4b6d-a42f-e8863058b58e" containerName="revoke-keys" Oct 30 21:14:43.248981 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.248959 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.255339 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.255300 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:14:43.257025 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.257003 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:14:43.257206 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.257181 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:14:43.257320 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.257239 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:14:43.258401 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.258377 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85"] Oct 30 21:14:43.352613 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.352567 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.352613 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.352623 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vxh\" (UniqueName: \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.352899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.352818 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.453606 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.453563 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.453830 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.453638 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.453830 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.453700 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vxh\" (UniqueName: \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.454281 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.454260 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.456226 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.456201 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.468293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.468252 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vxh\" (UniqueName: \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.563107 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.563074 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:43.838254 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.838228 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85"] Oct 30 21:14:43.840774 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:14:43.840742 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404868af_af8c_4544_923f_080362efa0a0.slice/crio-33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde WatchSource:0}: Error finding container 33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde: Status 404 returned error can't find the container with id 33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde Oct 30 21:14:43.874289 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:43.874257 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" event={"ID":"404868af-af8c-4544-923f-080362efa0a0","Type":"ContainerStarted","Data":"33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde"} Oct 30 21:14:44.879517 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:44.879428 5855 generic.go:356] "Generic (PLEG): container finished" podID="404868af-af8c-4544-923f-080362efa0a0" containerID="1c8d23dbbcff4ad79b6a638aeee551f101eb45a1ba428d4637d5b4ddc1b114ff" exitCode=0 Oct 30 21:14:44.879946 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:44.879513 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" event={"ID":"404868af-af8c-4544-923f-080362efa0a0","Type":"ContainerDied","Data":"1c8d23dbbcff4ad79b6a638aeee551f101eb45a1ba428d4637d5b4ddc1b114ff"} Oct 30 21:14:46.016899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.016866 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:46.079102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.079063 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\" (UniqueName: \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\") pod \"404868af-af8c-4544-923f-080362efa0a0\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " Oct 30 21:14:46.079102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.079103 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\") pod \"404868af-af8c-4544-923f-080362efa0a0\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " Oct 30 21:14:46.079384 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.079137 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\") pod \"404868af-af8c-4544-923f-080362efa0a0\" (UID: \"404868af-af8c-4544-923f-080362efa0a0\") " Oct 30 21:14:46.079569 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.079542 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca" pod "404868af-af8c-4544-923f-080362efa0a0" (UID: "404868af-af8c-4544-923f-080362efa0a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:46.081433 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.081406 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca" pod "404868af-af8c-4544-923f-080362efa0a0" (UID: "404868af-af8c-4544-923f-080362efa0a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:46.081518 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.081462 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh" pod "404868af-af8c-4544-923f-080362efa0a0" (UID: "404868af-af8c-4544-923f-080362efa0a0"). InnerVolumeSpecName "kube-api-access-57vxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:46.180460 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.180341 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vxh\" (UniqueName: \"kubernetes.io/projected/404868af-af8c-4544-923f-080362efa0a0-kube-api-access-57vxh\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.180460 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.180396 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404868af-af8c-4544-923f-080362efa0a0-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.180460 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.180409 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404868af-af8c-4544-923f-080362efa0a0-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:46.889491 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.889466 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" Oct 30 21:14:46.889713 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.889486 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85" event={"ID":"404868af-af8c-4544-923f-080362efa0a0","Type":"ContainerDied","Data":"33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde"} Oct 30 21:14:46.889713 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:46.889537 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b3c0b052e3d6ce96fe72e24330f3f2e314484072e604af4e82115818a0bcde" Oct 30 21:14:53.152162 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.152121 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" podUID="b0713a19-aeda-4e77-953a-398636610739" containerName="main" containerID="cri-o://ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6" gracePeriod=45 Oct 30 21:14:53.411959 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.411884 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:14:53.555876 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.555839 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\" (UniqueName: \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556072 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.555927 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556072 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.555994 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556072 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556015 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556072 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556039 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556072 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556066 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\") pod \"b0713a19-aeda-4e77-953a-398636610739\" (UID: \"b0713a19-aeda-4e77-953a-398636610739\") " Oct 30 21:14:53.556312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556252 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:53.556312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556282 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:53.556426 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.556394 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:53.558476 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.558442 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:53.558476 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.558458 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:53.558636 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.558477 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk" pod "b0713a19-aeda-4e77-953a-398636610739" (UID: "b0713a19-aeda-4e77-953a-398636610739"). InnerVolumeSpecName "kube-api-access-mn9pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:53.657181 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657142 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.657181 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657174 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.657407 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657203 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.657407 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657213 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0713a19-aeda-4e77-953a-398636610739-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.657407 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657223 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn9pk\" (UniqueName: \"kubernetes.io/projected/b0713a19-aeda-4e77-953a-398636610739-kube-api-access-mn9pk\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.657407 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.657242 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0713a19-aeda-4e77-953a-398636610739-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:53.924911 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.924866 5855 generic.go:356] "Generic (PLEG): container finished" podID="b0713a19-aeda-4e77-953a-398636610739" containerID="ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6" exitCode=0 Oct 30 21:14:53.925120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.924969 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" Oct 30 21:14:53.925120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.924984 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" event={"ID":"b0713a19-aeda-4e77-953a-398636610739","Type":"ContainerDied","Data":"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6"} Oct 30 21:14:53.925120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.925035 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds" event={"ID":"b0713a19-aeda-4e77-953a-398636610739","Type":"ContainerDied","Data":"1beb943c876628e32a18de2e8345fa55c6dbb9f845ce94b346756b4102061c07"} Oct 30 21:14:53.925120 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.925062 5855 scope.go:122] "RemoveContainer" containerID="ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6" Oct 30 21:14:53.938508 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.938481 5855 scope.go:122] "RemoveContainer" containerID="ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6" Oct 30 21:14:53.938927 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:14:53.938868 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6\": container with ID starting with ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6 not found: ID does not exist" containerID="ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6" Oct 30 21:14:53.938927 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.938903 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6"} err="failed to get container status \"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6\": rpc error: code = NotFound desc = could not find container \"ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6\": container with ID starting with ca828b728faeeaa0a4b8da29c86832a0aa31637a70f50823711e5fee7eaf2dc6 not found: ID does not exist" Oct 30 21:14:53.961565 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.961534 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:14:53.977246 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:53.977205 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-30d5ac/simulator-single-main-30d5ac-kserve-76774599fd-9cfds"] Oct 30 21:14:54.373675 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:54.373620 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85"] Oct 30 21:14:54.384752 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:54.384718 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-30d5ac-42n85"] Oct 30 21:14:54.776710 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:54.776611 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404868af-af8c-4544-923f-080362efa0a0" path="/var/lib/kubelet/pods/404868af-af8c-4544-923f-080362efa0a0/volumes" Oct 30 21:14:54.777063 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:54.777050 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0713a19-aeda-4e77-953a-398636610739" path="/var/lib/kubelet/pods/b0713a19-aeda-4e77-953a-398636610739/volumes" Oct 30 21:14:55.624007 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:14:55.623972 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:15:00.926445 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:15:00.926407 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:15:05.277213 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277178 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:15:05.277690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277510 5855 cpu_manager.go:414] "containerMap: removing container" podUID="404868af-af8c-4544-923f-080362efa0a0" containerName="revoke-keys" Oct 30 21:15:05.277690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277525 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404868af-af8c-4544-923f-080362efa0a0" containerName="revoke-keys" Oct 30 21:15:05.277690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277563 5855 cpu_manager.go:414] "containerMap: removing container" podUID="b0713a19-aeda-4e77-953a-398636610739" containerName="main" Oct 30 21:15:05.277690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277574 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0713a19-aeda-4e77-953a-398636610739" containerName="main" Oct 30 21:15:05.277690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277686 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="404868af-af8c-4544-923f-080362efa0a0" containerName="revoke-keys" Oct 30 21:15:05.277856 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.277705 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0713a19-aeda-4e77-953a-398636610739" containerName="main" Oct 30 21:15:05.282330 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.282302 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.293790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.291341 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:15:05.293790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.291445 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:15:05.293790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.292905 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c25rl\"" Oct 30 21:15:05.293790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.292922 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0cbb4221-kserve-self-signed-certs\"" Oct 30 21:15:05.293790 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.293067 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:15:05.370981 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.370936 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.370981 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.370983 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.371278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.371067 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.371278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.371109 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rcf\" (UniqueName: \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.371278 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.371146 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.371435 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.371329 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472134 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472065 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472134 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472148 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472569 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472183 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472569 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472206 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472569 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472372 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rcf\" (UniqueName: \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472569 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472453 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472823 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.472868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472845 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.473025 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.472995 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.475993 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.475945 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.476271 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.476242 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.489961 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.489915 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rcf\" (UniqueName: \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\") pod \"e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.606104 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.606051 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:05.882889 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.882863 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:15:05.885123 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:15:05.885095 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bca2197_c7c4_485b_bd51_41bfc0593e18.slice/crio-f34ecc3d32fdb791ab85bafd32420b4c05b0207da994dc1e77c9f779b4062ffa WatchSource:0}: Error finding container f34ecc3d32fdb791ab85bafd32420b4c05b0207da994dc1e77c9f779b4062ffa: Status 404 returned error can't find the container with id f34ecc3d32fdb791ab85bafd32420b4c05b0207da994dc1e77c9f779b4062ffa Oct 30 21:15:05.982437 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.982393 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" event={"ID":"0bca2197-c7c4-485b-bd51-41bfc0593e18","Type":"ContainerStarted","Data":"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2"} Oct 30 21:15:05.982606 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:05.982454 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" event={"ID":"0bca2197-c7c4-485b-bd51-41bfc0593e18","Type":"ContainerStarted","Data":"f34ecc3d32fdb791ab85bafd32420b4c05b0207da994dc1e77c9f779b4062ffa"} Oct 30 21:15:06.003330 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:06.003272 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" podStartSLOduration=1.003246425 podStartE2EDuration="1.003246425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:06.002046157 +0000 UTC m=+2119.360758905" watchObservedRunningTime="2026-10-30 21:15:06.003246425 +0000 UTC m=+2119.361959170" Oct 30 21:15:06.607223 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:06.607175 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:06.719680 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:06.719635 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:15:15.606510 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:15.606468 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:15.619791 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:15.619756 5855 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:16.050289 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:16.050260 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:15:24.229934 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:24.229892 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:15:30.924634 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:15:30.924594 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:15:39.294708 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:39.294648 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:15:59.921165 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:15:59.921066 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:16:00.926136 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:16:00.926094 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:16:01.223053 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:01.222949 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:16:03.962826 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:03.962784 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:03.963261 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:03.962844 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.543751 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:07.543701 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:16:14.942104 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:14.942056 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerName="main" containerID="cri-o://01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2" gracePeriod=45 Oct 30 21:16:15.039357 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.039311 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerName="main" probeResult="failure" output="Get \"https://10.134.0.149:8000/ready\": dial tcp 10.134.0.149:8000: connect: connection refused" Oct 30 21:16:15.194714 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.194633 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:16:15.218849 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.218819 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219044 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.218870 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219044 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.218937 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\" (UniqueName: \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219044 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.218972 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219044 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219037 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219285 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219072 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\") pod \"0bca2197-c7c4-485b-bd51-41bfc0593e18\" (UID: \"0bca2197-c7c4-485b-bd51-41bfc0593e18\") " Oct 30 21:16:15.219285 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219200 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:15.219433 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219392 5855 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-model-cache\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.219528 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219503 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:15.219589 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.219521 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:15.221607 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.221580 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:15.221748 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.221641 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:15.221748 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.221695 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf" pod "0bca2197-c7c4-485b-bd51-41bfc0593e18" (UID: "0bca2197-c7c4-485b-bd51-41bfc0593e18"). InnerVolumeSpecName "kube-api-access-24rcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:15.319979 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.319943 5855 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-home\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.319979 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.319973 5855 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bca2197-c7c4-485b-bd51-41bfc0593e18-tls-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.319979 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.319983 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rcf\" (UniqueName: \"kubernetes.io/projected/0bca2197-c7c4-485b-bd51-41bfc0593e18-kube-api-access-24rcf\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.320222 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.319993 5855 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-tmp-dir\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.320222 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.320001 5855 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bca2197-c7c4-485b-bd51-41bfc0593e18-dshm\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:15.381037 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.380961 5855 generic.go:356] "Generic (PLEG): container finished" podID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerID="01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2" exitCode=0 Oct 30 21:16:15.381410 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.381069 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" Oct 30 21:16:15.381410 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.381068 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" event={"ID":"0bca2197-c7c4-485b-bd51-41bfc0593e18","Type":"ContainerDied","Data":"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2"} Oct 30 21:16:15.381410 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.381165 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc" event={"ID":"0bca2197-c7c4-485b-bd51-41bfc0593e18","Type":"ContainerDied","Data":"f34ecc3d32fdb791ab85bafd32420b4c05b0207da994dc1e77c9f779b4062ffa"} Oct 30 21:16:15.381410 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.381202 5855 scope.go:122] "RemoveContainer" containerID="01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2" Oct 30 21:16:15.395862 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.395828 5855 scope.go:122] "RemoveContainer" containerID="01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2" Oct 30 21:16:15.396403 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:16:15.396371 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2\": container with ID starting with 01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2 not found: ID does not exist" containerID="01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2" Oct 30 21:16:15.396517 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.396425 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2"} err="failed to get container status \"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2\": rpc error: code = NotFound desc = could not find container \"01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2\": container with ID starting with 01014d44079818a5086e7b05b4d7a688174cb00f41f26f8ec82f076cea4fcdc2 not found: ID does not exist" Oct 30 21:16:15.406691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.406603 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:16:15.413032 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:15.412982 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0cbb4221-kserve-7b96bbc5fc-tscdc"] Oct 30 21:16:16.783272 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:16.783236 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" path="/var/lib/kubelet/pods/0bca2197-c7c4-485b-bd51-41bfc0593e18/volumes" Oct 30 21:16:17.303376 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:17.303341 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:16:18.967882 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:18.967838 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-npbcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:18.968304 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:18.967910 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:28.416374 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:28.416339 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:16:30.966948 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:16:30.966907 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:16:45.621921 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:45.621879 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:16:58.546205 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.546169 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:16:58.546791 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.546737 5855 cpu_manager.go:414] "containerMap: removing container" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerName="main" Oct 30 21:16:58.546791 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.546760 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerName="main" Oct 30 21:16:58.546927 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.546859 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bca2197-c7c4-485b-bd51-41bfc0593e18" containerName="main" Oct 30 21:16:58.551704 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.551642 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.552586 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.552560 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:16:58.555894 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.555873 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.564505 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.564470 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c3e6d92c\"" Oct 30 21:16:58.567945 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.567920 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:16:58.575528 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.575504 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqzn\" (UniqueName: \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.575671 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.575533 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.575742 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.575719 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdsj\" (UniqueName: \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.575801 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.575760 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.586273 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.586242 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:16:58.676547 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.676516 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdsj\" (UniqueName: \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.676547 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.676557 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.676812 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.676608 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqzn\" (UniqueName: \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.676812 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.676627 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.677338 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.677314 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.677404 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.677392 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.689427 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.689395 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqzn\" (UniqueName: \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\") pod \"payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.689591 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.689512 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdsj\" (UniqueName: \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\") pod \"payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:58.871564 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.871467 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:58.877734 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:58.877716 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:59.151312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.151279 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:16:59.153767 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:16:59.153711 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe0e1c3_fc34_421b_b429_1802c700636b.slice/crio-7e9930320697fe399a4af48f960cd028cfca9621cafd2dca5c5e37e1b6cb051d WatchSource:0}: Error finding container 7e9930320697fe399a4af48f960cd028cfca9621cafd2dca5c5e37e1b6cb051d: Status 404 returned error can't find the container with id 7e9930320697fe399a4af48f960cd028cfca9621cafd2dca5c5e37e1b6cb051d Oct 30 21:16:59.156685 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.156641 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:16:59.158833 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:16:59.158802 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928e00ed_82ce_4614_9721_7cd39c872e55.slice/crio-5b22909ed0863b87796a8f29c32b50e222068f6d99be2de8520b4ac972748a6c WatchSource:0}: Error finding container 5b22909ed0863b87796a8f29c32b50e222068f6d99be2de8520b4ac972748a6c: Status 404 returned error can't find the container with id 5b22909ed0863b87796a8f29c32b50e222068f6d99be2de8520b4ac972748a6c Oct 30 21:16:59.594782 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.594749 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" event={"ID":"dfe0e1c3-fc34-421b-b429-1802c700636b","Type":"ContainerStarted","Data":"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45"} Oct 30 21:16:59.595280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.594798 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" event={"ID":"dfe0e1c3-fc34-421b-b429-1802c700636b","Type":"ContainerStarted","Data":"7e9930320697fe399a4af48f960cd028cfca9621cafd2dca5c5e37e1b6cb051d"} Oct 30 21:16:59.595280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.594841 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:16:59.596353 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.596329 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" event={"ID":"928e00ed-82ce-4614-9721-7cd39c872e55","Type":"ContainerStarted","Data":"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5"} Oct 30 21:16:59.596478 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.596372 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" event={"ID":"928e00ed-82ce-4614-9721-7cd39c872e55","Type":"ContainerStarted","Data":"5b22909ed0863b87796a8f29c32b50e222068f6d99be2de8520b4ac972748a6c"} Oct 30 21:16:59.596478 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.596401 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:16:59.614848 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.614793 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" podStartSLOduration=1.614776203 podStartE2EDuration="1.614776203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:59.614211574 +0000 UTC m=+2232.972924319" watchObservedRunningTime="2026-10-30 21:16:59.614776203 +0000 UTC m=+2232.973488950" Oct 30 21:16:59.632860 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:16:59.632793 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" podStartSLOduration=1.6327741260000002 podStartE2EDuration="1.632774126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:59.632414544 +0000 UTC m=+2232.991127294" watchObservedRunningTime="2026-10-30 21:16:59.632774126 +0000 UTC m=+2232.991486870" Oct 30 21:17:00.707037 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:00.707001 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:17:00.924305 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:00.924261 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:17:10.603581 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:10.603551 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:17:10.604100 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:10.603622 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:17:13.259674 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.259631 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p"] Oct 30 21:17:13.262905 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.262886 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.270236 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.269897 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:17:13.271491 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.271456 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:17:13.271491 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.271474 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:17:13.271697 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.271571 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:17:13.273556 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.273533 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p"] Oct 30 21:17:13.326055 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.326021 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.326055 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.326066 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.326315 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.326159 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.427791 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.427749 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.427993 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.427812 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.427993 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.427878 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.428487 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.428468 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.430367 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:13.430342 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:13.434555 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:13.434535 5855 projected.go:196] Error preparing data for projected volume kube-api-access-dcd64 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 21:17:13.434683 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:13.434599 5855 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64 podName:a346a9ff-e943-4181-b45e-89dc31c98c70 nodeName:}" failed. No retries permitted until 2026-10-30 21:17:13.934583127 +0000 UTC m=+2247.293295851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dcd64" (UniqueName: "kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64") pod "maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" (UID: "a346a9ff-e943-4181-b45e-89dc31c98c70") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 21:17:14.035672 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.035626 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:14.053444 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.053410 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") pod \"maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:14.177177 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.177128 5855 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" secret="" err="secret \"maas-api-cleanup-dockercfg-7hf8r\" not found" Oct 30 21:17:14.177374 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.177213 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:14.451718 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.451687 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p"] Oct 30 21:17:14.453939 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:17:14.453908 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda346a9ff_e943_4181_b45e_89dc31c98c70.slice/crio-5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0 WatchSource:0}: Error finding container 5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0: Status 404 returned error can't find the container with id 5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0 Oct 30 21:17:14.667040 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:14.667001 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" event={"ID":"a346a9ff-e943-4181-b45e-89dc31c98c70","Type":"ContainerStarted","Data":"5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0"} Oct 30 21:17:15.674201 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:15.674095 5855 generic.go:356] "Generic (PLEG): container finished" podID="a346a9ff-e943-4181-b45e-89dc31c98c70" containerID="0fb510f6a547b0b8701bf3e3117622912de987844623a7006ad74a49a58289ff" exitCode=0 Oct 30 21:17:15.674685 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:15.674192 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" event={"ID":"a346a9ff-e943-4181-b45e-89dc31c98c70","Type":"ContainerDied","Data":"0fb510f6a547b0b8701bf3e3117622912de987844623a7006ad74a49a58289ff"} Oct 30 21:17:16.812896 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.812869 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:16.968928 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.968830 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\") pod \"a346a9ff-e943-4181-b45e-89dc31c98c70\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " Oct 30 21:17:16.969113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.968938 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") pod \"a346a9ff-e943-4181-b45e-89dc31c98c70\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " Oct 30 21:17:16.969113 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.968977 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\") pod \"a346a9ff-e943-4181-b45e-89dc31c98c70\" (UID: \"a346a9ff-e943-4181-b45e-89dc31c98c70\") " Oct 30 21:17:16.969390 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.969354 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca" pod "a346a9ff-e943-4181-b45e-89dc31c98c70" (UID: "a346a9ff-e943-4181-b45e-89dc31c98c70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:16.971104 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.971084 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca" pod "a346a9ff-e943-4181-b45e-89dc31c98c70" (UID: "a346a9ff-e943-4181-b45e-89dc31c98c70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:16.971207 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:16.971136 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64" pod "a346a9ff-e943-4181-b45e-89dc31c98c70" (UID: "a346a9ff-e943-4181-b45e-89dc31c98c70"). InnerVolumeSpecName "kube-api-access-dcd64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:17.069753 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.069712 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcd64\" (UniqueName: \"kubernetes.io/projected/a346a9ff-e943-4181-b45e-89dc31c98c70-kube-api-access-dcd64\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.069753 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.069748 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.069753 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.069761 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a346a9ff-e943-4181-b45e-89dc31c98c70-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:17.683848 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.683806 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" event={"ID":"a346a9ff-e943-4181-b45e-89dc31c98c70","Type":"ContainerDied","Data":"5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0"} Oct 30 21:17:17.683848 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.683825 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p" Oct 30 21:17:17.684075 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:17.683864 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c320582693f10713e3c229bfb015404c454fd1212ae8334921e1d2ddffbacd0" Oct 30 21:17:23.377494 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:23.377448 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:17:23.391744 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:23.391678 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:17:24.591025 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:24.590945 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p"] Oct 30 21:17:24.593853 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:24.593829 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c3e6d92c-5lt7p"] Oct 30 21:17:24.777968 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:24.777938 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a346a9ff-e943-4181-b45e-89dc31c98c70" path="/var/lib/kubelet/pods/a346a9ff-e943-4181-b45e-89dc31c98c70/volumes" Oct 30 21:17:25.935977 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:25.935943 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:17:28.378283 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.378213 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" podUID="928e00ed-82ce-4614-9721-7cd39c872e55" containerName="payload-processing" containerID="cri-o://055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5" gracePeriod=25 Oct 30 21:17:28.393088 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.393027 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" podUID="dfe0e1c3-fc34-421b-b429-1802c700636b" containerName="payload-pre-processing" containerID="cri-o://09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45" gracePeriod=25 Oct 30 21:17:28.641174 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.641113 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:17:28.672000 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.671978 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:17:28.733477 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.733438 5855 generic.go:356] "Generic (PLEG): container finished" podID="928e00ed-82ce-4614-9721-7cd39c872e55" containerID="055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5" exitCode=0 Oct 30 21:17:28.733690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.733484 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" Oct 30 21:17:28.733690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.733533 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" event={"ID":"928e00ed-82ce-4614-9721-7cd39c872e55","Type":"ContainerDied","Data":"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5"} Oct 30 21:17:28.733690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.733618 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47" event={"ID":"928e00ed-82ce-4614-9721-7cd39c872e55","Type":"ContainerDied","Data":"5b22909ed0863b87796a8f29c32b50e222068f6d99be2de8520b4ac972748a6c"} Oct 30 21:17:28.733863 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.733648 5855 scope.go:122] "RemoveContainer" containerID="055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5" Oct 30 21:17:28.734958 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.734937 5855 generic.go:356] "Generic (PLEG): container finished" podID="dfe0e1c3-fc34-421b-b429-1802c700636b" containerID="09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45" exitCode=0 Oct 30 21:17:28.735047 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.734987 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" event={"ID":"dfe0e1c3-fc34-421b-b429-1802c700636b","Type":"ContainerDied","Data":"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45"} Oct 30 21:17:28.735047 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.734993 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" Oct 30 21:17:28.735047 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.735017 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw" event={"ID":"dfe0e1c3-fc34-421b-b429-1802c700636b","Type":"ContainerDied","Data":"7e9930320697fe399a4af48f960cd028cfca9621cafd2dca5c5e37e1b6cb051d"} Oct 30 21:17:28.742409 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.742391 5855 scope.go:122] "RemoveContainer" containerID="055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5" Oct 30 21:17:28.742730 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:28.742711 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5\": container with ID starting with 055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5 not found: ID does not exist" containerID="055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5" Oct 30 21:17:28.742806 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.742738 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5"} err="failed to get container status \"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5\": rpc error: code = NotFound desc = could not find container \"055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5\": container with ID starting with 055e2ce598bd0a8391bf0e4331d7431053a10c61f438d4d3a48309be0d462ee5 not found: ID does not exist" Oct 30 21:17:28.742806 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.742755 5855 scope.go:122] "RemoveContainer" containerID="09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45" Oct 30 21:17:28.750502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.750485 5855 scope.go:122] "RemoveContainer" containerID="09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45" Oct 30 21:17:28.750792 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:28.750774 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45\": container with ID starting with 09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45 not found: ID does not exist" containerID="09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45" Oct 30 21:17:28.750848 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.750800 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45"} err="failed to get container status \"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45\": rpc error: code = NotFound desc = could not find container \"09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45\": container with ID starting with 09e0e976fd609ec02b884f8c3b088e7be9d36298611caa57276343a376c3fa45 not found: ID does not exist" Oct 30 21:17:28.783502 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.783474 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\" (UniqueName: \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\") pod \"dfe0e1c3-fc34-421b-b429-1802c700636b\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " Oct 30 21:17:28.783683 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.783575 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\") pod \"928e00ed-82ce-4614-9721-7cd39c872e55\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " Oct 30 21:17:28.783683 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.783629 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\") pod \"dfe0e1c3-fc34-421b-b429-1802c700636b\" (UID: \"dfe0e1c3-fc34-421b-b429-1802c700636b\") " Oct 30 21:17:28.783683 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.783649 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\" (UniqueName: \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\") pod \"928e00ed-82ce-4614-9721-7cd39c872e55\" (UID: \"928e00ed-82ce-4614-9721-7cd39c872e55\") " Oct 30 21:17:28.784007 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.783985 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume" pod "dfe0e1c3-fc34-421b-b429-1802c700636b" (UID: "dfe0e1c3-fc34-421b-b429-1802c700636b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:28.784079 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.784011 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume" pod "928e00ed-82ce-4614-9721-7cd39c872e55" (UID: "928e00ed-82ce-4614-9721-7cd39c872e55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:28.785918 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.785899 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn" pod "dfe0e1c3-fc34-421b-b429-1802c700636b" (UID: "dfe0e1c3-fc34-421b-b429-1802c700636b"). InnerVolumeSpecName "kube-api-access-bqqzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:28.785977 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.785926 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj" pod "928e00ed-82ce-4614-9721-7cd39c872e55" (UID: "928e00ed-82ce-4614-9721-7cd39c872e55"). InnerVolumeSpecName "kube-api-access-lbdsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:28.884839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.884803 5855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfe0e1c3-fc34-421b-b429-1802c700636b-plugins-config-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:28.884839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.884834 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdsj\" (UniqueName: \"kubernetes.io/projected/928e00ed-82ce-4614-9721-7cd39c872e55-kube-api-access-lbdsj\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:28.884839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.884843 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqqzn\" (UniqueName: \"kubernetes.io/projected/dfe0e1c3-fc34-421b-b429-1802c700636b-kube-api-access-bqqzn\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:28.885083 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:28.884852 5855 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/928e00ed-82ce-4614-9721-7cd39c872e55-plugins-config-volume\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:29.058649 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:29.058607 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:17:29.063135 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:29.063109 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c3e6d92c-676f7b796d-ldd47"] Oct 30 21:17:29.074942 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:29.074903 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:17:29.079039 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:29.079003 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c3e6d92c-65bf445b55-9qrgw"] Oct 30 21:17:30.777887 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:30.777844 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928e00ed-82ce-4614-9721-7cd39c872e55" path="/var/lib/kubelet/pods/928e00ed-82ce-4614-9721-7cd39c872e55/volumes" Oct 30 21:17:30.778414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:30.778379 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe0e1c3-fc34-421b-b429-1802c700636b" path="/var/lib/kubelet/pods/dfe0e1c3-fc34-421b-b429-1802c700636b/volumes" Oct 30 21:17:30.924169 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:17:30.924130 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:17:48.837797 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:17:48.837755 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:18:00.925197 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:18:00.925150 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:18:30.924863 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:18:30.924821 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:19:00.942106 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:19:00.942005 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:19:15.443238 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443202 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct"] Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443552 5855 cpu_manager.go:414] "containerMap: removing container" podUID="a346a9ff-e943-4181-b45e-89dc31c98c70" containerName="revoke-keys" Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443565 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a346a9ff-e943-4181-b45e-89dc31c98c70" containerName="revoke-keys" Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443587 5855 cpu_manager.go:414] "containerMap: removing container" podUID="dfe0e1c3-fc34-421b-b429-1802c700636b" containerName="payload-pre-processing" Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443602 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe0e1c3-fc34-421b-b429-1802c700636b" containerName="payload-pre-processing" Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443615 5855 cpu_manager.go:414] "containerMap: removing container" podUID="928e00ed-82ce-4614-9721-7cd39c872e55" containerName="payload-processing" Oct 30 21:19:15.443691 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443622 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928e00ed-82ce-4614-9721-7cd39c872e55" containerName="payload-processing" Oct 30 21:19:15.443966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443746 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="a346a9ff-e943-4181-b45e-89dc31c98c70" containerName="revoke-keys" Oct 30 21:19:15.443966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443761 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe0e1c3-fc34-421b-b429-1802c700636b" containerName="payload-pre-processing" Oct 30 21:19:15.443966 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.443770 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="928e00ed-82ce-4614-9721-7cd39c872e55" containerName="payload-processing" Oct 30 21:19:15.446348 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.446327 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.454209 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.453381 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9qkrz\"" Oct 30 21:19:15.454209 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.453455 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:19:15.455505 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.454970 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:19:15.455505 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.455251 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:19:15.456605 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.456583 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct"] Oct 30 21:19:15.512440 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.512392 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bccpd\" (UniqueName: \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.512629 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.512454 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.512629 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.512518 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.613263 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.613220 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.613446 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.613296 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bccpd\" (UniqueName: \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.613446 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.613329 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.614128 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.614106 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.615826 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.615803 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.625908 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.625882 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bccpd\" (UniqueName: \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:15.761406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:15.761312 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:16.038164 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:16.038105 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct"] Oct 30 21:19:16.040797 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:19:16.040768 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9912cdb7_b82f_4b16_bf46_c409dc176537.slice/crio-4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a WatchSource:0}: Error finding container 4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a: Status 404 returned error can't find the container with id 4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a Oct 30 21:19:16.234135 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:16.234094 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" event={"ID":"9912cdb7-b82f-4b16-bf46-c409dc176537","Type":"ContainerStarted","Data":"4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a"} Oct 30 21:19:17.240225 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:17.240118 5855 generic.go:356] "Generic (PLEG): container finished" podID="9912cdb7-b82f-4b16-bf46-c409dc176537" containerID="ecee000d75571a1ebf6b0eabf4db9dccc67de7efefcc4444d9b14666af7456e3" exitCode=0 Oct 30 21:19:17.240633 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:17.240209 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" event={"ID":"9912cdb7-b82f-4b16-bf46-c409dc176537","Type":"ContainerDied","Data":"ecee000d75571a1ebf6b0eabf4db9dccc67de7efefcc4444d9b14666af7456e3"} Oct 30 21:19:18.376251 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.376222 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:18.439583 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.439549 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\") pod \"9912cdb7-b82f-4b16-bf46-c409dc176537\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " Oct 30 21:19:18.439784 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.439599 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\" (UniqueName: \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\") pod \"9912cdb7-b82f-4b16-bf46-c409dc176537\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " Oct 30 21:19:18.439784 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.439637 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\") pod \"9912cdb7-b82f-4b16-bf46-c409dc176537\" (UID: \"9912cdb7-b82f-4b16-bf46-c409dc176537\") " Oct 30 21:19:18.439991 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.439969 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca" pod "9912cdb7-b82f-4b16-bf46-c409dc176537" (UID: "9912cdb7-b82f-4b16-bf46-c409dc176537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:18.441861 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.441836 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca" pod "9912cdb7-b82f-4b16-bf46-c409dc176537" (UID: "9912cdb7-b82f-4b16-bf46-c409dc176537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:18.441977 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.441914 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd" pod "9912cdb7-b82f-4b16-bf46-c409dc176537" (UID: "9912cdb7-b82f-4b16-bf46-c409dc176537"). InnerVolumeSpecName "kube-api-access-bccpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:18.540560 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.540469 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bccpd\" (UniqueName: \"kubernetes.io/projected/9912cdb7-b82f-4b16-bf46-c409dc176537-kube-api-access-bccpd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:18.540560 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.540503 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:18.540560 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:18.540516 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9912cdb7-b82f-4b16-bf46-c409dc176537-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:19.250175 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:19.250136 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" Oct 30 21:19:19.250334 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:19.250141 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct" event={"ID":"9912cdb7-b82f-4b16-bf46-c409dc176537","Type":"ContainerDied","Data":"4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a"} Oct 30 21:19:19.250334 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:19.250247 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b4d2d512e2da9d04e2e5c4010697f7765de554544237c28389e1830cfdad14a" Oct 30 21:19:25.034726 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:25.034679 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6"] Oct 30 21:19:25.037136 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:25.037109 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad14ae1-686x6"] Oct 30 21:19:26.778780 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:26.778744 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545a95c9-c8f1-4b6d-a42f-e8863058b58e" path="/var/lib/kubelet/pods/545a95c9-c8f1-4b6d-a42f-e8863058b58e/volumes" Oct 30 21:19:30.804331 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:30.804294 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct"] Oct 30 21:19:30.806913 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:30.806883 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-dd6b21-bq4ct"] Oct 30 21:19:30.930950 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:19:30.930904 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:19:32.186868 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:32.186820 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:19:32.777559 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:32.777522 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9912cdb7-b82f-4b16-bf46-c409dc176537" path="/var/lib/kubelet/pods/9912cdb7-b82f-4b16-bf46-c409dc176537/volumes" Oct 30 21:19:44.443367 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:44.443309 5855 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal" podUID="f7ada5c68d422440bbf7fd2285220de0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 21:19:49.798736 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:49.798689 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:19:54.991958 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:54.991924 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:19:59.347487 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.347439 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn"] Oct 30 21:19:59.348059 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.348004 5855 cpu_manager.go:414] "containerMap: removing container" podUID="9912cdb7-b82f-4b16-bf46-c409dc176537" containerName="revoke-keys" Oct 30 21:19:59.348059 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.348023 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9912cdb7-b82f-4b16-bf46-c409dc176537" containerName="revoke-keys" Oct 30 21:19:59.348189 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.348104 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="9912cdb7-b82f-4b16-bf46-c409dc176537" containerName="revoke-keys" Oct 30 21:19:59.351456 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.351433 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.359773 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.359744 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:19:59.360061 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.359913 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9qkrz\"" Oct 30 21:19:59.360061 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.359952 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:19:59.360061 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.359972 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn"] Oct 30 21:19:59.360061 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.360003 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:19:59.466404 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.466365 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.466592 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.466433 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.466592 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.466513 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbbz\" (UniqueName: \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.567673 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.567622 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.567881 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.567715 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.567881 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.567744 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbbz\" (UniqueName: \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.568388 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.568362 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.570154 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.570134 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.580594 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.580568 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbbz\" (UniqueName: \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\") pod \"maas-api-revoke-sub-f03629db-snvvn\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.667464 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.667380 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:19:59.940866 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:19:59.940837 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn"] Oct 30 21:19:59.943850 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:19:59.943823 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80aef8b_396a_45a0_8eec_978b0295af6c.slice/crio-dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8 WatchSource:0}: Error finding container dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8: Status 404 returned error can't find the container with id dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8 Oct 30 21:20:00.492333 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:00.492294 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" event={"ID":"e80aef8b-396a-45a0-8eec-978b0295af6c","Type":"ContainerStarted","Data":"26c1c3e993fcab0dee3d4f2e62ec788493ed403c23479dec1ef79e4607623eae"} Oct 30 21:20:00.492784 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:00.492376 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" event={"ID":"e80aef8b-396a-45a0-8eec-978b0295af6c","Type":"ContainerStarted","Data":"dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8"} Oct 30 21:20:00.511300 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:00.511237 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" podStartSLOduration=1.511221567 podStartE2EDuration="1.511221567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:00.511031378 +0000 UTC m=+2413.869744128" watchObservedRunningTime="2026-10-30 21:20:00.511221567 +0000 UTC m=+2413.869934313" Oct 30 21:20:00.924746 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:20:00.924710 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:20:01.498507 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:01.498471 5855 generic.go:356] "Generic (PLEG): container finished" podID="e80aef8b-396a-45a0-8eec-978b0295af6c" containerID="26c1c3e993fcab0dee3d4f2e62ec788493ed403c23479dec1ef79e4607623eae" exitCode=0 Oct 30 21:20:01.498945 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:01.498558 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" event={"ID":"e80aef8b-396a-45a0-8eec-978b0295af6c","Type":"ContainerDied","Data":"26c1c3e993fcab0dee3d4f2e62ec788493ed403c23479dec1ef79e4607623eae"} Oct 30 21:20:02.651648 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.651616 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:20:02.694110 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.694074 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\" (UniqueName: \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\") pod \"e80aef8b-396a-45a0-8eec-978b0295af6c\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " Oct 30 21:20:02.694321 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.694158 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\") pod \"e80aef8b-396a-45a0-8eec-978b0295af6c\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " Oct 30 21:20:02.694321 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.694179 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\") pod \"e80aef8b-396a-45a0-8eec-978b0295af6c\" (UID: \"e80aef8b-396a-45a0-8eec-978b0295af6c\") " Oct 30 21:20:02.694647 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.694623 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca" pod "e80aef8b-396a-45a0-8eec-978b0295af6c" (UID: "e80aef8b-396a-45a0-8eec-978b0295af6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:02.696547 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.696520 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca" pod "e80aef8b-396a-45a0-8eec-978b0295af6c" (UID: "e80aef8b-396a-45a0-8eec-978b0295af6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:02.696635 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.696527 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz" pod "e80aef8b-396a-45a0-8eec-978b0295af6c" (UID: "e80aef8b-396a-45a0-8eec-978b0295af6c"). InnerVolumeSpecName "kube-api-access-kbbbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:02.795194 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.795157 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:02.795194 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.795194 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80aef8b-396a-45a0-8eec-978b0295af6c-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:02.795503 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:02.795207 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbbbz\" (UniqueName: \"kubernetes.io/projected/e80aef8b-396a-45a0-8eec-978b0295af6c-kube-api-access-kbbbz\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:03.508934 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:03.508903 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" Oct 30 21:20:03.508934 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:03.508924 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn" event={"ID":"e80aef8b-396a-45a0-8eec-978b0295af6c","Type":"ContainerDied","Data":"dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8"} Oct 30 21:20:03.509324 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:03.508970 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd76817d08d313b5c0893ad5d3b55d31e6c392747fa260ac20fd026f2d9c5b8" Oct 30 21:20:10.122417 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:10.122380 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:20:15.296395 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.296358 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf"] Oct 30 21:20:15.296839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.296711 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e80aef8b-396a-45a0-8eec-978b0295af6c" containerName="revoke-keys" Oct 30 21:20:15.296839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.296723 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80aef8b-396a-45a0-8eec-978b0295af6c" containerName="revoke-keys" Oct 30 21:20:15.296839 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.296790 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80aef8b-396a-45a0-8eec-978b0295af6c" containerName="revoke-keys" Oct 30 21:20:15.299650 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.299630 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.308921 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.308882 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9qkrz\"" Oct 30 21:20:15.309385 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.308903 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:20:15.309385 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.309343 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:20:15.309907 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.309886 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:20:15.311198 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.311173 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf"] Oct 30 21:20:15.353743 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.353707 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqzp\" (UniqueName: \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.353947 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.353787 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.353947 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.353835 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.455374 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.455327 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.455576 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.455397 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.455576 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.455437 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqzp\" (UniqueName: \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.456116 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.456081 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.458004 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.457974 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.468762 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.468737 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqzp\" (UniqueName: \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\") pod \"maas-api-revoke-sub-0165d7d2-sncsf\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.614485 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.614394 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:15.894613 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:15.894585 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf"] Oct 30 21:20:15.897410 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:20:15.897382 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e60d0c_c0e2_4296_bb18_311c5c0f082c.slice/crio-2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d WatchSource:0}: Error finding container 2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d: Status 404 returned error can't find the container with id 2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d Oct 30 21:20:16.573521 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:16.573481 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" event={"ID":"89e60d0c-c0e2-4296-bb18-311c5c0f082c","Type":"ContainerStarted","Data":"53fb81f1f7cfcacc7669ffabd544fbf7a618a6be4cf8dc125b34d05c982cc619"} Oct 30 21:20:16.573979 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:16.573553 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" event={"ID":"89e60d0c-c0e2-4296-bb18-311c5c0f082c","Type":"ContainerStarted","Data":"2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d"} Oct 30 21:20:16.594191 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:16.594134 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" podStartSLOduration=1.594118216 podStartE2EDuration="1.594118216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.592552418 +0000 UTC m=+2429.951265160" watchObservedRunningTime="2026-10-30 21:20:16.594118216 +0000 UTC m=+2429.952830959" Oct 30 21:20:17.580280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:17.580245 5855 generic.go:356] "Generic (PLEG): container finished" podID="89e60d0c-c0e2-4296-bb18-311c5c0f082c" containerID="53fb81f1f7cfcacc7669ffabd544fbf7a618a6be4cf8dc125b34d05c982cc619" exitCode=0 Oct 30 21:20:17.580760 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:17.580334 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" event={"ID":"89e60d0c-c0e2-4296-bb18-311c5c0f082c","Type":"ContainerDied","Data":"53fb81f1f7cfcacc7669ffabd544fbf7a618a6be4cf8dc125b34d05c982cc619"} Oct 30 21:20:18.715952 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.715920 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:18.788081 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.788026 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\") pod \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " Oct 30 21:20:18.788081 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.788084 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\") pod \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " Oct 30 21:20:18.788338 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.788131 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\" (UniqueName: \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\") pod \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\" (UID: \"89e60d0c-c0e2-4296-bb18-311c5c0f082c\") " Oct 30 21:20:18.788524 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.788495 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca" pod "89e60d0c-c0e2-4296-bb18-311c5c0f082c" (UID: "89e60d0c-c0e2-4296-bb18-311c5c0f082c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:18.790312 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.790283 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca" pod "89e60d0c-c0e2-4296-bb18-311c5c0f082c" (UID: "89e60d0c-c0e2-4296-bb18-311c5c0f082c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:18.790529 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.790508 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp" pod "89e60d0c-c0e2-4296-bb18-311c5c0f082c" (UID: "89e60d0c-c0e2-4296-bb18-311c5c0f082c"). InnerVolumeSpecName "kube-api-access-jmqzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:18.889285 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.889200 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:18.889285 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.889231 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e60d0c-c0e2-4296-bb18-311c5c0f082c-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:18.889285 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:18.889246 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmqzp\" (UniqueName: \"kubernetes.io/projected/89e60d0c-c0e2-4296-bb18-311c5c0f082c-kube-api-access-jmqzp\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:19.589740 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:19.589703 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" Oct 30 21:20:19.589949 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:19.589708 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf" event={"ID":"89e60d0c-c0e2-4296-bb18-311c5c0f082c","Type":"ContainerDied","Data":"2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d"} Oct 30 21:20:19.589949 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:19.589847 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b35f3c29de77006015b1163939c8a26d4594c7c761d20c6c278bbaced82551d" Oct 30 21:20:26.043395 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.043357 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6"] Oct 30 21:20:26.044003 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.043981 5855 cpu_manager.go:414] "containerMap: removing container" podUID="89e60d0c-c0e2-4296-bb18-311c5c0f082c" containerName="revoke-keys" Oct 30 21:20:26.044052 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.044007 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e60d0c-c0e2-4296-bb18-311c5c0f082c" containerName="revoke-keys" Oct 30 21:20:26.044117 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.044106 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e60d0c-c0e2-4296-bb18-311c5c0f082c" containerName="revoke-keys" Oct 30 21:20:26.047279 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.047257 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.052818 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.052795 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:20:26.053159 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.053119 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9qkrz\"" Oct 30 21:20:26.054284 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.054243 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:20:26.054742 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.054465 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:20:26.056175 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.056113 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6"] Oct 30 21:20:26.159761 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.159722 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.159761 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.159762 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.160041 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.159795 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzsgb\" (UniqueName: \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.261393 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.261351 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.261393 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.261390 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.261733 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.261422 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzsgb\" (UniqueName: \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.262109 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.262088 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.264074 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.264048 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.275069 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.275040 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzsgb\" (UniqueName: \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\") pod \"maas-api-revoke-sub-6b19338f-977z6\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.364557 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.364521 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:26.657575 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:26.657469 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6"] Oct 30 21:20:26.662896 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:20:26.662817 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62d22ec_750c_4dc4_b7ac_a7fef30889e3.slice/crio-b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096 WatchSource:0}: Error finding container b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096: Status 404 returned error can't find the container with id b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096 Oct 30 21:20:27.628959 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:27.628924 5855 generic.go:356] "Generic (PLEG): container finished" podID="e62d22ec-750c-4dc4-b7ac-a7fef30889e3" containerID="ebc57b7da2316c532df30435505f97822b73ebfdd1249c8374ab74f179796f4a" exitCode=0 Oct 30 21:20:27.629373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:27.629033 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" event={"ID":"e62d22ec-750c-4dc4-b7ac-a7fef30889e3","Type":"ContainerDied","Data":"ebc57b7da2316c532df30435505f97822b73ebfdd1249c8374ab74f179796f4a"} Oct 30 21:20:27.629373 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:27.629074 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" event={"ID":"e62d22ec-750c-4dc4-b7ac-a7fef30889e3","Type":"ContainerStarted","Data":"b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096"} Oct 30 21:20:28.769585 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.769559 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:28.889214 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.889116 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\") pod \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " Oct 30 21:20:28.889404 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.889285 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\" (UniqueName: \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\") pod \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " Oct 30 21:20:28.889404 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.889332 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\") pod \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\" (UID: \"e62d22ec-750c-4dc4-b7ac-a7fef30889e3\") " Oct 30 21:20:28.889548 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.889518 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca" pod "e62d22ec-750c-4dc4-b7ac-a7fef30889e3" (UID: "e62d22ec-750c-4dc4-b7ac-a7fef30889e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:28.889742 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.889717 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-openshift-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:28.891725 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.891692 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca" pod "e62d22ec-750c-4dc4-b7ac-a7fef30889e3" (UID: "e62d22ec-750c-4dc4-b7ac-a7fef30889e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:28.891811 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.891720 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb" pod "e62d22ec-750c-4dc4-b7ac-a7fef30889e3" (UID: "e62d22ec-750c-4dc4-b7ac-a7fef30889e3"). InnerVolumeSpecName "kube-api-access-kzsgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:28.990250 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.990200 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzsgb\" (UniqueName: \"kubernetes.io/projected/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-kube-api-access-kzsgb\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:28.990250 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:28.990243 5855 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62d22ec-750c-4dc4-b7ac-a7fef30889e3-maas-api-xks-ca\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:20:29.639414 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:29.639387 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" Oct 30 21:20:29.639597 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:29.639391 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6" event={"ID":"e62d22ec-750c-4dc4-b7ac-a7fef30889e3","Type":"ContainerDied","Data":"b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096"} Oct 30 21:20:29.639597 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:29.639500 5855 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7f8686364045d70dbb419eba27049453bdfec0fbfcd8efeea275d3c2fbb6096" Oct 30 21:20:30.923050 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:20:30.923012 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:20:36.830808 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:36.830774 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tccqk"] Oct 30 21:20:48.546770 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:48.546739 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdjpb_6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30/manager/0.log" Oct 30 21:20:50.567693 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:50.567647 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sncsf_89e60d0c-c0e2-4296-bb18-311c5c0f082c/revoke-keys/0.log" Oct 30 21:20:50.915144 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:50.914997 5855 scope.go:122] "RemoveContainer" containerID="000a7ab0a553a11ff483419c7be11358dc5bff3f7ef5455a439e26e588e17e34" Oct 30 21:20:50.923443 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:50.923422 5855 scope.go:122] "RemoveContainer" containerID="1c8d23dbbcff4ad79b6a638aeee551f101eb45a1ba428d4637d5b4ddc1b114ff" Oct 30 21:20:51.300568 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:51.300536 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-977z6_e62d22ec-750c-4dc4-b7ac-a7fef30889e3/revoke-keys/0.log" Oct 30 21:20:51.468450 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:51.468419 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-snvvn_e80aef8b-396a-45a0-8eec-978b0295af6c/revoke-keys/0.log" Oct 30 21:20:54.205515 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:54.205483 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-tccqk_c28881ee-b9e8-42fc-9ff5-da0fffce60ec/limitador/0.log" Oct 30 21:20:55.351768 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:55.351737 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-xvsjm_7f4ed90c-75e2-457c-9d45-53a1a27b25fb/discovery/0.log" Oct 30 21:20:55.714614 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:55.714513 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c6fcc99c4-lnchp_f388c115-91a5-4c47-91fb-cb53bf16fe59/kube-auth-proxy/0.log" Oct 30 21:20:56.813581 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:56.813548 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-npbcs_03bdc55d-6e24-4211-9244-e23b02f1d2fb/istio-proxy/0.log" Oct 30 21:20:56.992333 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:56.992301 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ph82z_6b19e8fb-247e-4f4e-850b-60074ef710ab/istio-proxy/0.log" Oct 30 21:20:57.171273 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:20:57.171143 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-spcjv_180b2492-c057-4886-89f9-d748be169507/istio-proxy/0.log" Oct 30 21:21:00.923913 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:21:00.923872 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:21:11.194762 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:11.194715 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdjpb_6f8a22c7-ddeb-4e31-8cab-54d6e40a9c30/manager/0.log" Oct 30 21:21:12.938690 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:12.938615 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-sncsf_89e60d0c-c0e2-4296-bb18-311c5c0f082c/revoke-keys/0.log" Oct 30 21:21:13.292083 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:13.292005 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-977z6_e62d22ec-750c-4dc4-b7ac-a7fef30889e3/revoke-keys/0.log" Oct 30 21:21:13.474451 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:13.474381 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-snvvn_e80aef8b-396a-45a0-8eec-978b0295af6c/revoke-keys/0.log" Oct 30 21:21:25.802471 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:25.802441 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-tccqk_c28881ee-b9e8-42fc-9ff5-da0fffce60ec/limitador/0.log" Oct 30 21:21:26.771318 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:26.771278 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-xvsjm_7f4ed90c-75e2-457c-9d45-53a1a27b25fb/discovery/0.log" Oct 30 21:21:27.144386 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:27.144352 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c6fcc99c4-lnchp_f388c115-91a5-4c47-91fb-cb53bf16fe59/kube-auth-proxy/0.log" Oct 30 21:21:28.263302 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:28.263273 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-npbcs_03bdc55d-6e24-4211-9244-e23b02f1d2fb/istio-proxy/0.log" Oct 30 21:21:28.446386 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:28.446352 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-ph82z_6b19e8fb-247e-4f4e-850b-60074ef710ab/istio-proxy/0.log" Oct 30 21:21:28.624104 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:21:28.624078 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-spcjv_180b2492-c057-4886-89f9-d748be169507/istio-proxy/0.log" Oct 30 21:21:30.923645 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:21:30.923596 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:22:00.923387 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:22:00.923297 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:22:30.925611 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:22:30.925561 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:23:00.923987 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:23:00.923950 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:23:30.923862 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:23:30.923779 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:23:50.957617 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:23:50.957583 5855 scope.go:122] "RemoveContainer" containerID="0fb510f6a547b0b8701bf3e3117622912de987844623a7006ad74a49a58289ff" Oct 30 21:24:00.923860 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:24:00.923813 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:24:30.924953 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:24:30.924916 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:25:00.923166 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:25:00.923078 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:25:03.021787 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:03.021750 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn"] Oct 30 21:25:03.024369 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:03.024340 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-snvvn"] Oct 30 21:25:04.776904 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:04.776870 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80aef8b-396a-45a0-8eec-978b0295af6c" path="/var/lib/kubelet/pods/e80aef8b-396a-45a0-8eec-978b0295af6c/volumes" Oct 30 21:25:11.204187 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.204151 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hgxj/must-gather-q8tsd"] Oct 30 21:25:11.204701 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.204510 5855 cpu_manager.go:414] "containerMap: removing container" podUID="e62d22ec-750c-4dc4-b7ac-a7fef30889e3" containerName="revoke-keys" Oct 30 21:25:11.204701 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.204524 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62d22ec-750c-4dc4-b7ac-a7fef30889e3" containerName="revoke-keys" Oct 30 21:25:11.204701 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.204601 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62d22ec-750c-4dc4-b7ac-a7fef30889e3" containerName="revoke-keys" Oct 30 21:25:11.207884 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.207866 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.213034 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.213014 5855 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hgxj\"/\"default-dockercfg-z7szz\"" Oct 30 21:25:11.213174 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.213063 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hgxj\"/\"openshift-service-ca.crt\"" Oct 30 21:25:11.213174 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.213081 5855 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hgxj\"/\"kube-root-ca.crt\"" Oct 30 21:25:11.231379 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.231352 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/must-gather-q8tsd"] Oct 30 21:25:11.301757 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.301714 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27262399-dbe4-4279-a0bc-ba5a7649ad7e-must-gather-output\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.301929 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.301779 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sgz\" (UniqueName: \"kubernetes.io/projected/27262399-dbe4-4279-a0bc-ba5a7649ad7e-kube-api-access-x5sgz\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.402622 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.402579 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27262399-dbe4-4279-a0bc-ba5a7649ad7e-must-gather-output\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.402622 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.402630 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sgz\" (UniqueName: \"kubernetes.io/projected/27262399-dbe4-4279-a0bc-ba5a7649ad7e-kube-api-access-x5sgz\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.403047 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.403022 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/27262399-dbe4-4279-a0bc-ba5a7649ad7e-must-gather-output\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.415297 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.415263 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sgz\" (UniqueName: \"kubernetes.io/projected/27262399-dbe4-4279-a0bc-ba5a7649ad7e-kube-api-access-x5sgz\") pod \"must-gather-q8tsd\" (UID: \"27262399-dbe4-4279-a0bc-ba5a7649ad7e\") " pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.532750 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.532618 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" Oct 30 21:25:11.812150 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.812072 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/must-gather-q8tsd"] Oct 30 21:25:11.815136 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:25:11.815109 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27262399_dbe4_4279_a0bc_ba5a7649ad7e.slice/crio-4be392e985e5c37deefd46bdb44b387b7130d746b38d138514b7bd05b3700c4b WatchSource:0}: Error finding container 4be392e985e5c37deefd46bdb44b387b7130d746b38d138514b7bd05b3700c4b: Status 404 returned error can't find the container with id 4be392e985e5c37deefd46bdb44b387b7130d746b38d138514b7bd05b3700c4b Oct 30 21:25:11.817060 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.817043 5855 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:25:11.950517 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:11.950485 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" event={"ID":"27262399-dbe4-4279-a0bc-ba5a7649ad7e","Type":"ContainerStarted","Data":"4be392e985e5c37deefd46bdb44b387b7130d746b38d138514b7bd05b3700c4b"} Oct 30 21:25:12.958338 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:12.958306 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" event={"ID":"27262399-dbe4-4279-a0bc-ba5a7649ad7e","Type":"ContainerStarted","Data":"c3759c578a8a5a7625038fdeb8b07831349409ecde63b8ee32a853e803352fff"} Oct 30 21:25:13.967492 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:13.967448 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" event={"ID":"27262399-dbe4-4279-a0bc-ba5a7649ad7e","Type":"ContainerStarted","Data":"c5a82b2057f071274b3537c260bb2de5fee9e8612a06eebde04cfb6049e5ede5"} Oct 30 21:25:13.989129 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:13.989074 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2hgxj/must-gather-q8tsd" podStartSLOduration=2.083672041 podStartE2EDuration="2.989059073s" totalImagesPullingTime="905.387032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:13.987332992 +0000 UTC m=+2727.346045737" watchObservedRunningTime="2026-10-30 21:25:13.989059073 +0000 UTC m=+2727.347771819" Oct 30 21:25:14.769803 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:14.769772 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wnpsx_fc36e1bc-d1aa-4f15-bb34-5971c8c2b74b/global-pull-secret-syncer/0.log" Oct 30 21:25:15.062811 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:15.062772 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ll9wf_8bd09df0-d50b-4f00-863d-276efe60e695/konnectivity-agent/0.log" Oct 30 21:25:15.160899 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:15.160857 5855 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-65.us-east-2.compute.internal_f7ada5c68d422440bbf7fd2285220de0/haproxy/0.log" Oct 30 21:25:19.023260 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:19.023222 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf"] Oct 30 21:25:19.025355 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:19.025327 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-sncsf"] Oct 30 21:25:20.777445 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:20.777411 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e60d0c-c0e2-4296-bb18-311c5c0f082c" path="/var/lib/kubelet/pods/89e60d0c-c0e2-4296-bb18-311c5c0f082c/volumes" Oct 30 21:25:29.024240 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:29.024201 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6"] Oct 30 21:25:29.026588 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:29.026560 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-977z6"] Oct 30 21:25:30.777541 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:30.777505 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62d22ec-750c-4dc4-b7ac-a7fef30889e3" path="/var/lib/kubelet/pods/e62d22ec-750c-4dc4-b7ac-a7fef30889e3/volumes" Oct 30 21:25:30.923374 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:25:30.923334 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:25:50.983592 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:50.983461 5855 scope.go:122] "RemoveContainer" containerID="ecee000d75571a1ebf6b0eabf4db9dccc67de7efefcc4444d9b14666af7456e3" Oct 30 21:25:53.849392 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.848527 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 21:25:53.849392 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.848834 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" containerID="cri-o://eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d" gracePeriod=30 Oct 30 21:25:53.850833 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.850804 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 21:25:53.851095 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.851057 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" containerID="cri-o://7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f" gracePeriod=30 Oct 30 21:25:53.852246 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.852223 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 21:25:53.852472 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:53.852429 5855 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" containerID="cri-o://9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21" gracePeriod=30 Oct 30 21:25:59.120393 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.120307 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187184 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187245 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187294 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187321 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187367 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.187406 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187402 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.189065 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187435 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.189065 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187495 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.189065 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.187527 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\") pod \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\" (UID: \"03bdc55d-6e24-4211-9244-e23b02f1d2fb\") " Oct 30 21:25:59.189065 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.188430 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.189774 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.189065 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.189919 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.189876 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.191418 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.190713 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.193073 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.192798 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.193735 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.193646 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.194280 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.194253 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "kube-api-access-6vtp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.195212 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.195191 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.196062 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.196040 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token" pod "03bdc55d-6e24-4211-9244-e23b02f1d2fb" (UID: "03bdc55d-6e24-4211-9244-e23b02f1d2fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.198870 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.198833 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 21:25:59.201353 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.201329 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 21:25:59.222559 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.222530 5855 generic.go:356] "Generic (PLEG): container finished" podID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerID="eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d" exitCode=0 Oct 30 21:25:59.222841 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.222582 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" Oct 30 21:25:59.222841 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.222631 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" event={"ID":"03bdc55d-6e24-4211-9244-e23b02f1d2fb","Type":"ContainerDied","Data":"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d"} Oct 30 21:25:59.222841 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.222718 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs" event={"ID":"03bdc55d-6e24-4211-9244-e23b02f1d2fb","Type":"ContainerDied","Data":"f88bd33ebbcb0883a95778a238962f3d7be36617d3e3cb60225ad9c17e89212f"} Oct 30 21:25:59.222841 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.222756 5855 scope.go:122] "RemoveContainer" containerID="eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d" Oct 30 21:25:59.231787 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.231574 5855 generic.go:356] "Generic (PLEG): container finished" podID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerID="7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f" exitCode=0 Oct 30 21:25:59.232104 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.232072 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" event={"ID":"6b19e8fb-247e-4f4e-850b-60074ef710ab","Type":"ContainerDied","Data":"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f"} Oct 30 21:25:59.232220 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.232140 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" event={"ID":"6b19e8fb-247e-4f4e-850b-60074ef710ab","Type":"ContainerDied","Data":"9899e15dc01c7740955c27df7e2ea477393c1b92b526be6d3ec6990b2f2c2754"} Oct 30 21:25:59.232220 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.232205 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z" Oct 30 21:25:59.235362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.235254 5855 generic.go:356] "Generic (PLEG): container finished" podID="180b2492-c057-4886-89f9-d748be169507" containerID="9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21" exitCode=0 Oct 30 21:25:59.235362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.235288 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" event={"ID":"180b2492-c057-4886-89f9-d748be169507","Type":"ContainerDied","Data":"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21"} Oct 30 21:25:59.235362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.235322 5855 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" Oct 30 21:25:59.235362 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.235332 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" event={"ID":"180b2492-c057-4886-89f9-d748be169507","Type":"ContainerDied","Data":"0ced0ff21126487597361b7f5337f1c597debb1de9fd341de2d9853a7253734d"} Oct 30 21:25:59.244806 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.244783 5855 scope.go:122] "RemoveContainer" containerID="eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d" Oct 30 21:25:59.245241 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:25:59.245216 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d\": container with ID starting with eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d not found: ID does not exist" containerID="eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d" Oct 30 21:25:59.245340 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.245255 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d"} err="failed to get container status \"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d\": rpc error: code = NotFound desc = could not find container \"eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d\": container with ID starting with eda7590d6d7efeff14432a787fbf812ccc89f2e410ae0fc19fdc063ff25ac24d not found: ID does not exist" Oct 30 21:25:59.245340 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.245279 5855 scope.go:122] "RemoveContainer" containerID="7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f" Oct 30 21:25:59.260029 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.256683 5855 scope.go:122] "RemoveContainer" containerID="7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f" Oct 30 21:25:59.260029 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:25:59.257108 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f\": container with ID starting with 7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f not found: ID does not exist" containerID="7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f" Oct 30 21:25:59.260029 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.257145 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f"} err="failed to get container status \"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f\": rpc error: code = NotFound desc = could not find container \"7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f\": container with ID starting with 7e261308b4c5b28d1e8658c28051e2034cf6de93f6c70f74d30580e7e4d8398f not found: ID does not exist" Oct 30 21:25:59.260029 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.257171 5855 scope.go:122] "RemoveContainer" containerID="9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21" Oct 30 21:25:59.276307 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.276281 5855 scope.go:122] "RemoveContainer" containerID="9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21" Oct 30 21:25:59.276762 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:25:59.276696 5855 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21\": container with ID starting with 9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21 not found: ID does not exist" containerID="9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21" Oct 30 21:25:59.276762 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.276738 5855 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21"} err="failed to get container status \"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21\": rpc error: code = NotFound desc = could not find container \"9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21\": container with ID starting with 9275154e46ea6794cb78596145fbb893a5a78238bd47df2144ed9b60928cfe21 not found: ID does not exist" Oct 30 21:25:59.288772 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288743 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.288925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288796 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.288925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288827 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.288925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288861 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.288925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288888 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.288925 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.288922 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.289201 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289007 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.289262 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289222 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.289347 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289314 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.289400 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289376 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.289532 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289395 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.289532 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289410 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.289532 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289421 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.289532 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289443 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.289910 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.289649 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.290736 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290714 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.290856 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290773 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.290856 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290826 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\") pod \"6b19e8fb-247e-4f4e-850b-60074ef710ab\" (UID: \"6b19e8fb-247e-4f4e-850b-60074ef710ab\") " Oct 30 21:25:59.290975 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290872 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.290975 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290937 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.291091 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.290972 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.291091 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291006 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.291091 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291042 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.291246 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291089 5855 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\") pod \"180b2492-c057-4886-89f9-d748be169507\" (UID: \"180b2492-c057-4886-89f9-d748be169507\") " Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291404 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291427 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291443 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291456 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291470 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291484 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291498 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtp7\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-kube-api-access-6vtp7\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291512 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291525 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291537 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291553 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291569 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b19e8fb-247e-4f4e-850b-60074ef710ab-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291582 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291596 5855 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bdc55d-6e24-4211-9244-e23b02f1d2fb-credential-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291609 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291824 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.291914 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.292060 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.292102 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.292077 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.293081 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.292705 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 21:25:59.293322 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.293292 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.293837 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.293811 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "kube-api-access-58zgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.294965 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.294945 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.295831 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.295808 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.295962 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.295930 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token" pod "180b2492-c057-4886-89f9-d748be169507" (UID: "180b2492-c057-4886-89f9-d748be169507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.296127 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.296107 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.296193 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.296144 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.296307 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.296288 5855 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs" pod "6b19e8fb-247e-4f4e-850b-60074ef710ab" (UID: "6b19e8fb-247e-4f4e-850b-60074ef710ab"). InnerVolumeSpecName "kube-api-access-gh9vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.296447 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.296431 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-npbcs"] Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392640 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392690 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zgd\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-kube-api-access-58zgd\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392704 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392713 5855 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-envoy\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392721 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392730 5855 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9vs\" (UniqueName: \"kubernetes.io/projected/6b19e8fb-247e-4f4e-850b-60074ef710ab-kube-api-access-gh9vs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392738 5855 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/180b2492-c057-4886-89f9-d748be169507-istio-token\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.392746 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392747 5855 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/180b2492-c057-4886-89f9-d748be169507-istiod-ca-cert\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.393141 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392757 5855 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/180b2492-c057-4886-89f9-d748be169507-istio-podinfo\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.393141 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392767 5855 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-istio-data\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.393141 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392775 5855 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-certs\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.393141 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.392783 5855 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/180b2492-c057-4886-89f9-d748be169507-workload-socket\") on node \"ip-10-0-141-65.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.558992 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.558961 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 21:25:59.562707 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.562669 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-ph82z"] Oct 30 21:25:59.584059 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.583780 5855 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 21:25:59.585337 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:25:59.585300 5855 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv"] Oct 30 21:26:00.097907 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:00.097864 5855 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-spcjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:00.098117 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:00.097979 5855 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-spcjv" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:00.778140 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:00.778106 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" path="/var/lib/kubelet/pods/03bdc55d-6e24-4211-9244-e23b02f1d2fb/volumes" Oct 30 21:26:00.778528 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:00.778522 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="180b2492-c057-4886-89f9-d748be169507" path="/var/lib/kubelet/pods/180b2492-c057-4886-89f9-d748be169507/volumes" Oct 30 21:26:00.778939 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:00.778924 5855 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" path="/var/lib/kubelet/pods/6b19e8fb-247e-4f4e-850b-60074ef710ab/volumes" Oct 30 21:26:00.924063 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:26:00.924028 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:26:11.083795 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.083761 5855 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh"] Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084079 5855 cpu_manager.go:414] "containerMap: removing container" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084094 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084111 5855 cpu_manager.go:414] "containerMap: removing container" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084117 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084132 5855 cpu_manager.go:414] "containerMap: removing container" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084140 5855 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084225 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b19e8fb-247e-4f4e-850b-60074ef710ab" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084234 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="180b2492-c057-4886-89f9-d748be169507" containerName="istio-proxy" Oct 30 21:26:11.084293 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.084241 5855 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bdc55d-6e24-4211-9244-e23b02f1d2fb" containerName="istio-proxy" Oct 30 21:26:11.090518 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.090480 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.095829 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.095800 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh"] Oct 30 21:26:11.203200 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.203161 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-sys\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.203200 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.203198 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-podres\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.203449 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.203305 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8sm\" (UniqueName: \"kubernetes.io/projected/b9dce20a-9f46-41e6-a705-2ad0c8186564-kube-api-access-2x8sm\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.203449 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.203371 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-lib-modules\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.203449 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.203412 5855 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-proc\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304434 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-proc\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304470 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304473 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-sys\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304489 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-podres\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304541 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8sm\" (UniqueName: \"kubernetes.io/projected/b9dce20a-9f46-41e6-a705-2ad0c8186564-kube-api-access-2x8sm\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304573 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-sys\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304574 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-proc\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304569 5855 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-lib-modules\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304646 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-lib-modules\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.304793 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.304642 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b9dce20a-9f46-41e6-a705-2ad0c8186564-podres\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.317570 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.317540 5855 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8sm\" (UniqueName: \"kubernetes.io/projected/b9dce20a-9f46-41e6-a705-2ad0c8186564-kube-api-access-2x8sm\") pod \"perf-node-gather-daemonset-gmhrh\" (UID: \"b9dce20a-9f46-41e6-a705-2ad0c8186564\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.405344 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.405246 5855 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:11.679498 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:11.679469 5855 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh"] Oct 30 21:26:11.682203 ip-10-0-141-65 kubenswrapper[5855]: W1030 21:26:11.682165 5855 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb9dce20a_9f46_41e6_a705_2ad0c8186564.slice/crio-82f203f6a94c89161a06d1bfd1d6a628eedbeb0a975734ae94797c6967b5c9b5 WatchSource:0}: Error finding container 82f203f6a94c89161a06d1bfd1d6a628eedbeb0a975734ae94797c6967b5c9b5: Status 404 returned error can't find the container with id 82f203f6a94c89161a06d1bfd1d6a628eedbeb0a975734ae94797c6967b5c9b5 Oct 30 21:26:12.298667 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:12.298624 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" event={"ID":"b9dce20a-9f46-41e6-a705-2ad0c8186564","Type":"ContainerStarted","Data":"bd3ef524791118a7ae6bb96c7a6fd86437b6e9a1f3c667d2f775e6dc54bc28fb"} Oct 30 21:26:12.299111 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:12.298703 5855 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" event={"ID":"b9dce20a-9f46-41e6-a705-2ad0c8186564","Type":"ContainerStarted","Data":"82f203f6a94c89161a06d1bfd1d6a628eedbeb0a975734ae94797c6967b5c9b5"} Oct 30 21:26:12.299111 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:12.298802 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:12.317843 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:12.317781 5855 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" podStartSLOduration=1.317762273 podStartE2EDuration="1.317762273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:12.317620629 +0000 UTC m=+2785.676333375" watchObservedRunningTime="2026-10-30 21:26:12.317762273 +0000 UTC m=+2785.676475025" Oct 30 21:26:18.313841 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:18.313810 5855 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-gmhrh" Oct 30 21:26:30.924454 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:26:30.924364 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor" Oct 30 21:26:51.005897 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:51.005859 5855 scope.go:122] "RemoveContainer" containerID="ebc57b7da2316c532df30435505f97822b73ebfdd1249c8374ab74f179796f4a" Oct 30 21:26:51.014345 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:51.014320 5855 scope.go:122] "RemoveContainer" containerID="26c1c3e993fcab0dee3d4f2e62ec788493ed403c23479dec1ef79e4607623eae" Oct 30 21:26:51.022493 ip-10-0-141-65 kubenswrapper[5855]: I1030 21:26:51.022471 5855 scope.go:122] "RemoveContainer" containerID="53fb81f1f7cfcacc7669ffabd544fbf7a618a6be4cf8dc125b34d05c982cc619" Oct 30 21:27:00.923309 ip-10-0-141-65 kubenswrapper[5855]: E1030 21:27:00.923269 5855 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wm9wk" containerName="extractor"